To help novice signers learn American Sign Language, we develop PopSignAI, a proof-of-concept smartphone-based bubble-shooter game that facilitates real-time interaction through isolated sign language recognition. In a 20-person user study, we demonstrate that encouraging novice signers to practice generating sign in PopSignAI is more efficient for teaching ASL skills than a version of PopSign focused on receptive signing ability. We use over 200,000 examples of 250 signs from 47 signers to train and test a user-independent LSTM recognizer that achieves 82.9% accuracy on an independent test set. For the purposes of the game, the recognizer averages 99.6% accuracy with a 7ms inference time using a 2.5MB model. Ablation studies suggest that as few as eight signers are need for training in order for adequate recognition accuracy for PopSignAI’s gameplay. To encourage future sign language recognition games, we release the PopSignAI recognition pipeline and software. We identify hearing parents of deaf children as important potential users of sign games and conduct interviews with eight of these parents, investigating their motivation and challenges in learning sign.

Progress in machine understanding of sign languages has been slow and hampered by limited data. In this paper, we present FSboard, an American Sign Language finger-spelling dataset situated in a mobile text entry use case, collected from 147 paid and consenting Deaf signers using Pixel 4A selfie cameras in a variety of environments. Fin-gerspelling recognition is an incomplete solution that comprises only a small part of sign language translation, but it could provide some immediate benefit to Deaf/Hard of Hearing signers while more broadly capable technology develops. At >3 million characters in length and >250 hours in duration, FSboard is the largest fingerspelling recognition dataset to date by a factor of >10x. As a simple baseline, we finetune 30 Hz MediaPipe Holistic landmark inputs into ByT5-Small and achieve 11.1% Character Error Rate (CER) on a test set with unique phrases and signers. This quality degrades gracefully when decreasing frame rate and excluding face/body landmarks—plausible optimizations to help with on-device performance—but falls short of human performance measured at 2.2% CER.1

This study investigates the consistency and fairness of grading in higher education by analyzing the impact of individual graders on class averages, identifying grading inconsistencies through similarity assessments of essays, and examining the correlation between the use of detailed grading rubrics and the frequency of student re-grade requests. Utilizing a comprehensive dataset of student essays, assigned grades, and grading rubrics, the research employs statistical analyses and natural language processing techniques to address these objectives at scale. Using the central limit theorem, we can identify significant differences in overall scoring between graders. Additionally, the study reveals instances where textually similar essays received markedly different grades, highlighting potential inconsistencies in grading practices. Furthermore, our analysis reveals that graders who apply a larger number of rubric items in their grading tend to receive higher number of student regrade requests (r>0.87). This indicates that while rubric-based grading is intended to enhance transparency, its extensive use may leave room for subjective interpretation, leading to more frequent grade challenges. These insights contribute to the ongoing discourse on improving grading practices and offer recommendations for enhancing transparency and equity in academic assessments at scale.

We introduce “Whirling Interface,” a selection method for XR displays using bare-hand motion matching gestures as an input technique. We extend the motion matching input method, by introducing different input states to provide visual feedback and guidance to the users. Using the wrist joint as the primary input modality, our technique reduces user fatigue and improves performance while selecting small and distant targets. In a study with 16 participants, we compared the whirling interface with a standard ray casting method using hand gestures. The results demonstrate that the Whirling Interface consistently achieves high success rates, especially for distant targets, averaging 95.58% with a completion time of 5.58 seconds. Notably, it requires a smaller camera sensing field of view of only 21.45° horizontally and 24.7° vertically. Participants reported lower workloads on distant conditions and expressed a higher preference for the Whirling Interface in general. These findings suggest that the Whirling Interface could be a useful alternative input method for XR displays with a small camera sensing FOV or when interacting with small targets.

In this half-day workshop, we seek to establish a continuing dialogue across classrooms about the challenges faced from a rapidly expanding student body, such as is happening in Computer Science departments. We plan to catalog a list of these challenges, the tools developed internally to overcome these challenges, and the barriers to adoption of these tools for the learning community at large. We invite practitioners who are in the process or creating or adapting educational technologies for scale to share their challenges and experiences.

Cheating detection in large classes with online, take-home exams is an extremely difficult problem. As class size increases, the process of detection and evidence building becomes a significant investment of time. To identify cheating without invasive real-time monitoring, Examinator v4.0 uses answer rarity and submission timestamps. Creative question design allows detection of cheating even when answers are correct. Automatic report generation saves instructors time when compiling evidence for cases. Examinator v4.0 streamlines the identification and reporting of cheating students in online take-home exams, analyzing 7683 unique submissions from 1923 students across two semesters and resulting in 52 convictions of academic misconduct.

Cheating detection in large classes with online, take-home exams is an extremely difficult problem. While some cheating can be identified through statistical analysis of all student responses, this analysis can easily be fooled by "smart cheaters'' actively attempting to hide evidence of their unauthorized collaboration. We demonstrate the effectiveness of watermarks combined with creative question design to provide evidence of cheating. We provide results from an initial deployment of our answer watermarking method and do a case study into how "smart cheaters'' attempt to cover their tracks, demonstrating the need for more advanced methods of catching cheating in online, take-home exams.

Interactive teaching methods often lead to higher levels of student engagement with course material. Yet, as class sizes increase, the demand on teaching staff becomes unsustainable. Our solution, Socratic Mind, employs Large Language Models to provide scalable, interactive oral assessments by functioning as a virtual instructor. This paper discusses the outcomes and user feedback from the preliminary implementation of our system in a large classroom environment with 600 students.

The proliferation of powerful large language models with human-like abilities, like ChatGPT, pose serious challenges for educators to enforce academic integrity policies. To address this problem, we propose a novel approach that uses past students' essay submissions dated before the popularization of ChatGPT, and ChatGPT generated essay responses as ground truth to train classifiers to detect ChatGPT usage for current student submissions. Our case study found that, for the same question prompt, student written answers and ChatGPT generated answers are very different. Testing on the ground truth data shows very simple machine learning methods, including multinomial naive Bayes, linear discriminant analysis, and logistic regression, can achieve close to perfect accuracies in detecting ChatGPT generated responses. Using this approach, we suspect around 7% of current student submissions are ChatGPT generated.

Camera-based text entry using American Sign Language (ASL) fingerspelling has become more feasible due to recent advancements in recognition technology. However, there are numerous situations where camera-based text entry may not be ideal or acceptable. To address this, we present FingerSpeller, a solution that enables camera-free text entry using smart rings. FingerSpeller utilizes accelerometers embedded in five smart rings from TapStrap, a commercially available wearable keyboard, to track finger motion and recognize fingerspelling. A Hidden Markov Model (HMM) based backend with continuous Gaussian modeling facilitates accurate recognition as evaluated in a real-world deployment. In offline isolated word recognition experiments conducted on a 1,164-word dictionary, FingerSpeller achieves an average character accuracy of 91% and word accuracy of 87% across three participants. Furthermore, we demonstrate that the system can be downsized to only two rings while maintaining an accuracy level of approximately 90% compared to the original configuration. This reduction in form factor enhances user comfort and significantly improves the overall usability of the system.

Examinator v3.0 detects cheating in online take-home exams by comparing answers and the timestamps they were entered. A web interface enables efficient manual inspection. Use of the tool reveals that certain question types substantially enhance cheating detection, demonstrating the potential of automated algorithmic detection at scale. Examinator v3.0 has analyzed 915,831 pairs of exam submissions across three courses over two semesters at a top U.S. institution, identifying 46 instances of cheating.

As enrollment increases, teaching assistants (TAs) need help prioritizing responding to students' posts in on-line forums. In a graduate-level Artificial Intelligence forum, three instructors rank the urgency of posts which is compared to the ratings of 13 course TAs; correlation with the instructors' scores have an r=0.55, with a TA inter-rater reliability of 35%. However, when TAs used a codebook containing seven dimensions created by the instructors to define urgency levels, correlation increased to r=0.73 and reliability to 53%. The instructor rankings are also compared to cognitive presence ratings from the Community of Inquiry framework. Cognitive presence correlates with urgency with r=-0.68. These results suggest that recommendation agents that prioritize posts based on cognitive presence or the urgency codebook may be beneficial for TAs.

PopSign is a smartphone-based bubble-shooter game that helps hearing parentsof deaf infants learn sign language. To help parents practice their ability to sign,PopSign is integrating sign language recognition as part of its gameplay. Fortraining the recognizer, we introduce the PopSign ASL v1.0 dataset that collectsexamples of 250 isolated American Sign Language (ASL) signs using Pixel 4Asmartphone selfie cameras in a variety of environments. It is the largest publiclyavailable, isolated sign dataset by number of examples and is the first dataset tofocus on one-handed, smartphone signs. We collected over 210,000 examplesat 1944x2592 resolution made by 47 consenting Deaf adult signers for whomAmerican Sign Language is their primary language. We manually reviewed 217,866of these examples, of which 175,023 (approximately 700 per sign) were the signintended for the educational game. 39,304 examples were recognizable as a signbut were not the desired variant or were a different sign. We provide a training setof 31 signers, a validation set of eight signers, and a test set of eight signers. Abaseline LSTM model for the 250-sign vocabulary achieves 82.1% accuracy (81.9%class-weighted F1 score) on the validation set and 84.2% (83.9% class-weightedF1 score) on the test set. Gameplay suggests that accuracy will be sufficient forcreating educational games involving sign language recognition.

Soon, smartphones may be capable of allowing American Sign Language (ASL) signing and/or fingerspelling for text entry. To explore the usefulness of this approach, we compared emulated fingerspelling recognition with a virtual keyboard for 12 Deaf participants. With practice, fingerspelling is faster (42.5 wpm), potentially has fewer errors (4.02% corrected error rate) and higher throughput (14.2 bits/second), and is as desired as virtual keyboard texting (31.9 wpm; 6.46% corrected error rate; 10.9 bits/second throughput). Our second study recruits another 12 Deaf users at the 2022 National Association for the Deaf conference to compare the walk-up usability of fingerspelling alone, signing, and virtual keyboard text entry for interacting with an emulated mobile assistant. Both signing and virtual keyboard text entry were preferred over fingerspelling.

Speech is inappropriate in many situations, limiting when voice control can be used. Most unvoiced speech text entry systems can not be used while on-the-go due to movement artifacts. Using a dental retainer with capacitive touch sensors, SilentSpeller tracks tongue movement, enabling users to type by spelling words without voicing. SilentSpeller achieves an average 97% character accuracy in offline isolated word testing on a 1164-word dictionary. Walking has little effect on accuracy; average offline character accuracy was roughly equivalent on 107 phrases entered while walking (97.5%) or seated (96.5%). To demonstrate extensibility, the system was tested on 100 unseen words, leading to an average 94% accuracy. Live text entry speeds for seven participants averaged 37 words per minute at 87% accuracy. Comparing silent spelling to current practice suggests that SilentSpeller may be a viable alternative for silent mobile text entry.

Embedding a post recommendation system in online course forums improves students' ability to find relevant course content. Yet, there is limited research quantifying how these interventions impact forum interactions and students' class performance. We randomly divide an on-line class in a Masters degree-seeking program into control and experimental sections. Midway through the class, for the experimental section, we introduce an information retrieval system that suggests relevant posts to students while they compose their new posts. The average number of initial posts in the discussion forum dropped by 55% for the experimental group compared to 21% in the control group. In post-hoc analysis, we classify students as having an active or passive (i.e., mostly lurking) forum participation style. Posts per student per assignment by passive participants in the experimental group decreased by 15%, while passive participants' posts in the control group increased by 200%. The number of answers given by instructors in the experimental group decreased twice as much as the control group after intervention, though the difference was not statistically significant. The reduction of posts was not associated with a decrease in academic performance (grades) for the experimental group. This line of research convinced the popular forum Piazza to implement a post recommendation system similar to that used in these experiments.

Plagiarism in computer science programs remains a problem at many universities. Programming assignments lend themselves to unpermitted collaboration: students can share code or find solutions from past semesters through simple internet searches. JackMarker helps address this problem by embedding a hidden, traceable, and unique token within documents. GitDown extends this work by locating available online solutions and requesting that they be removed.

Augmenting everyday surfaces with interaction sensing capability that is maintenance-free, low-cost (∼ $1), and in an appropriate form factor is a challenge with current technologies. MARS (Multi-channel Ambiently-powered Realtime Sensing) enables battery-free sensing and wireless communication of touch, swipe, and speech interactions by combining a nanowatt programmable oscillator with frequency-shifted analog backscatter communication. A zero-threshold voltage field-effect transistor (FET) is used to create an oscillator with a low startup voltage (∼ 500 mV) and current (< 2uA), whose frequency can be affected through changes in inductance or capacitance from the user interactions. Multiple MARS systems can operate in the same environment by tuning each oscillator circuit to a different frequency range. The nanowatt power budget allows the system to be powered directly through ambient energy sources like photodiodes or thermoelectric generators. We differentiate MARS from previous systems based on power requirements, cost, and part count and explore different interaction and activity sensing scenarios suitable for indoor environments.

Bluetooth requires device pairing to ensure security in data transmission, encumbering a number of ad-hoc, transactional interactions that require both ease-of-use and “good enough” security: e.g., sharing contact information or secure links to people nearby. We introduce Bit Whisperer, an ad-hoc short-range wireless communication system that enables “walk up and share” data transmissions with “good enough” security. Bit Whisperer transmits data to proximate devices co-located on a solid surface through high frequency, inaudible acoustic signals. The physical surface has two benefits: it enhances acoustic signal transmission by reflecting sound waves as they propagate; and, it makes the domain of communication visible, helping users identify exactly with whom they are sharing data without prior pairing. Through a series of technical evaluations, we demonstrate that Bit Whisperer is robust for common use-cases and secure against likely threats. We also implement three example applications to demonstrate the utility of Whisperer: 1-to-1 local contact sharing, 1-to-N private link sharing to open a secure group chat, and 1-to-N local device authentication.

E-textile microinteractions advance cord-based interfaces by enabling the simultaneous use of precise continuous control and casual discrete gestures. We leverage the recently introduced I/O Braid sensing architecture to enable a series of user studies and experiments which help design suitable interactions and a real-time gesture recognition pipeline. Informed by a gesture elicitation study with 36 participants, we developed a user-dependent classifier for eight discrete gestures with 94% accuracy for 12 participants. In a formal evaluation we show that we can enable precise manipulation with the same architecture. Our quantitative targeting experiment suggests that twisting is faster than existing headphone button controls and is comparable in speed to a capacitive touch surface. Qualitative interview feedback indicates a preference for I/O Braid's interaction over that of in-line headphone controls. Our applications demonstrate how continuous and discrete gestures can be combined to form new, integrated e-textile microinteraction techniques for real-time continuous control, discrete actions and mode switching.

Research in dolphin communication and cognition requires detailed inspection of audible dolphin signals. The manual analysis of these signals is cumbersome and time-consuming. We seek to automate parts of the analysis using modern deep learning methods. We propose to learn an autoencoder constructed from convolutional and recurrent layers trained in an unsupervised fashion. The resulting model embeds patterns in audible dolphin communication. In several experiments, we show that the embeddings can be used for clustering as well as signal detection and signal type classification.

Examinator compares pairs of take-home exams to select which should be manually checked for plagiarism. Examinator also generates a report with evidence for these cases using its metrics and those generated as a by-product of the commercial grading tool Gradescope. Examinator supports degree-seeking graduate programs (both online and on-campus) at a top computer science graduate institute in the United States. Since Spring 2019, Examinator has compared over 2 million pairs of exams from a popular Artificial Intelligence course, resulting in 56 cases being referred for discipline. Iterative development has improved the percentage of referrals of suggested cases from 15% to 25%.

As enrollment numbers in online courses increase, students, instructors, and teaching assistants have difficulty finding needed information in online forums because of the number of posts, resulting in duplicate posts that exacerbate the problem. We introduce PARQR, a recommendation tool that suggests relevant contributions as participants compose their posts. We investigate the use of PARQR in five online degree-seeking courses. We survey 74 students and interview five teaching assistants to understand their experience with online forums and PARQR. We compare the differences between using and not using PARQR for an online course assignment. PARQR users found the tool to be useful for navigating online forums, and PARQR was effective in reducing the number of posts (0.291 vs. 0.506 posts per active student) and duplicate posts (17.8% vs. 25.6%) in an online course. These results suggest that PARQR makes on-line forums more efficient for users to find needed information.

As online educational programs scale, monitoring peer collaboration in platforms like BlueJeans for plagiarism becomes difficult. Recent studies indicate that students are less likely to cheat if presented with direct warning messages prior to engaging in online activities. In this work, we present Bluejeans codE Leak deTection (BELT), a system that monitors online BlueJeans meetings for shared code and sends timely warning messages to meeting participants. To test BELT's robustness as an online proctor, we evaluate its code-text disambiguation, code detection from images of varying quality, and code detection from videos of varying resolution. We conclude this work by pinpointing areas of improvement and briefly discuss possible extensions for future work.

Mobile solutions can help transform speech and sound into visual representations for people who are deaf or hard-of-hearing (DHH). However, where handheld phones present challenges, head-worn displays (HWDs) could further communication through privately transcribed text, hands-free use, improved mobility, and socially acceptable interactions.

We introduce Serpentine, a self-powered sensor that is a reversibly deformable cord capable of sensing a variety of human input. The material properties and structural design of Serpentine allow it to be flexible, twistable, stretchable and squeezable, enabling a broad variety of expressive input modalities. The sensor operates using the principle of Triboelectric Nanogenerators (TENG), which allows it to sense mechanical deformation without an external power source. The affordances of the cord include six interactions---Pluck, Twirl, Stretch, Pinch, Wiggle and Twist. Serpentine demonstrates the ability to simultaneously recognize these inputs through a single physical interface. A 12-participant user study illustrates 95.7% accuracy for a user-dependent recognition model using a realtime system and 92.17% for user-independent offline detection. We conclude by demonstrating how Serpentine can be employed in everyday ubiquitous computing applications.

JacquardTooIkit enables full access to the Levi's Jacquard Jacket and the ability to create gestures such as our new "Force Touch" gesture. We conduct a 25 participant user study testing all available gestures for accuracy and intuitiveness. "Brush In" and "Brush Out" are rated highly intuitive and have high recognition accuracy.

SelfSync enables rapid, robust initiation of a gesture interface using synchronized movement of different body parts. SelfSync is the gestural equivalent of a hotword such as OK-Google in a speech interface and is enabled by the increasing trend where a user wears two or more wearables, such as a smartwatch, wireless earbuds, or a smartphone. In a user study comparing five potential SelfSync gestures in isolation, our system averages 96%, 98% and 88% for user dependent, user adapted, and user independent accuracy, respectively. For when the user has a phone in a pocket and a smart-watch, we suggest twisting the hand about the wrist while moving the leg with the phone in synchrony left and right. When the user has a head worn device and a smartwatch, we suggest twisting the hand while twisting the head left and right.

We introduce PARQR, a tool for online education forums that reduces duplicate posts by 40% in a degree seeking online masters program at a top university. Instead of performing a standard keyword search, PARQR monitors questions as students compose them and continuously suggests relevant posts. In testing, PARQR correctly recommends a relevant post, if one exists, 73.5% of the time. We discuss PARQR's design, initial experimental results comparing different semesters with and without PARQR, and interviews we conducted with teaching instructors regarding their experience with PARQR.

Cheating has always been a problem for academic institutions, but the internet has increased access to a form of academic dishonesty known as contract cheating, or "homework for hire." When students purchase work online and submit it as their own, it cannot be detected by commonly-used plagiarism detection tools, and this troubling form of cheating seems to be increasing.

We present Surface++, which leverages our previous work SATURN, a self-powered flexible acoustic sensor, and ZEUSSS, a passive wireless sound communication technique using analog backscatter, to create a scalable and self-sustainable wireless sound sensing surface. Our new prototype allows for large area acoustic sensing using modular fabrication techniques with the promise of being fully printable. A single small Surface++ patch can be used to extend voice and gesture input for everyday surfaces, while our more sensitive Surface++ modular array allows for large-area context sensing and localization.

FingerPing is a novel sensing technique that can recognize various fine-grained hand poses by analyzing acoustic resonance features. A surface-transducer mounted on a thumb ring injects acoustic chirps (20Hz to 6,000Hz) to the body. Four receivers distributed on the wrist and thumb collect the chirps. Different hand poses of the hand create distinct paths for the acoustic chirps to travel, creating unique frequency responses at the four receivers. We demonstrate how FingerPing can differentiate up to 22 hand poses, including the thumb touching each of the 12 phalanges on the hand as well as 10 American sign language poses. A user study with 16 participants showed that our system can recognize these two sets of poses with an accuracy of 93.77% and 95.64%, respectively. We discuss the opportunities and remaining challenges for the widespread use of this input technique.

Washing hands is one of the easiest yet most effective ways to prevent spreading illnesses and diseases. However, not adhering to thorough handwashing routines is a substantial problem worldwide. For example, in hospital operations lack of hygiene leads to healthcare associated infections. We present WristWash, a wrist-worn sensing platform that integrates an inertial measurement unit and a Hidden Markov Model-based analysis method that enables automated assessments of handwashing routines according to recommendations provided by the World Health Organization (WHO). We evaluated Wrist-Wash in a case study with 12 participants. WristWash is able to successfully recognize the 13 steps of the WHO handwashing procedure with an average accuracy of 92% with user-dependent models, and with 85% for user-independent modeling. We further explored the system's robustness by conducting another case study with six participants, this time in an unconstrained environment, to test variations in the hand-washing routine and to show the potential for real-world deployments.

We extend the interaction space of low-cost mobile virtual reality (VR) by introducing bidirectional scrolling and discrete selection using magnetic sensing. Our design uses the original Google Cardboard v1 input components, modifying only the cardboard mounted on the side. Users slide the magnetized washer around a circular track on the outer layer, which drags a magnet on the inner layer across asymmetric patterned ridges. The phone's magnetometer detects the position of the magnet as it moves around the track and slots into each ridge, emulating a click wheel. The phone's accelerometer is used to recognize center button taps. We compare our system against the current best practice (gaze) with 12 participants across four VR navigation and selection tasks. Finally, we demonstrate our system robustly handles continuous input, despite some minor deterioration of the cardboard, using a motorized rig over an 8-hour period.

We describe the development of a system for recognizing dolphin whistles on the CHAT (cetacean hearing and telemetry) wearable underwater computer system. An Nvidia Jetson TK1 single board computer was installed in the existing chat systems to improve processing power and overall system power efficiency. The inclusion of a GPU allowed the system to recognize in real time dolphin whistles varied in both pitch and time by using a 192khz Fast Fourier Transform for spectral analysis, linear convolution filters for pattern extraction, and dynamic time warping for pattern recognition.

Haptic technology can be used as a tool for learning. Can even the haptic elements in a smartwatch teach a new skill? Here we present a case of using a smartwatch for passive tactile learning. We use the Sony Smartwatch 3 to teach users Morse code while they wear the watch but focus on unrelated tasks. An initial hypothesis forecasted that the stimulation from the smartwatch, typically used for message alerts, would be too subtle to enable haptic learning; however, we find significant improvements in six participants using the technique. Furthermore, we expose participants to two different durations of stimulation and find different results.

Order picking accounts for 55% of the annual $60 billion spent on warehouse operations in the United States. Reducing human-induced errors in the order fulfillment process can save warehouses and distributors significant costs. We investigate a radio-frequency identification (RFID)-based verification method wherein wearable RFID scanners, worn on the wrists, scan passive RFID tags mounted on an item's bin as the item is picked; this method is used in conjunction with a head-up display (HUD) to guide the user to the correct item. We compare this RFID verification method to pick-to-light with button verification, pick-to-paper with barcode verification, and pick-to-paper with no verification. We find that pick-to-HUD with RFID verification enables significantly faster picking, provides the lowest error rate, and provides the lowest task workload.

We present SeeSaw, a synchronous gesture interface for commodity smartwatches to support watch-hand only input with no additional hardware. Our algorithm, which uses correlation to determine whether the user is rotating their wrist in synchrony with a tactile and visual prompt, minimizes false-trigger events while maintaining fast input during situational impairments. Results from a 12 person evaluation of the system, used to respond to notifications on the watch during walking and simulated driving, show interaction speeds of 4.0 s - 5.5 s, which is comparable to the swipe-based interface control condition. SeeSaw is also evaluated as an input interface for watches used in conjunction with a head-worn display. A six subject study showed a 95% success rate in dismissing notifications and a 3.57 s mean dismissal time.

We introduce I/O Braid, an interactive textile cord with embedded sensing and visual feedback. I/O Braid senses proximity, touch, and twist through a spiraling, repeating braiding topology of touch matrices. This sensing topology is uniquely scalable, requiring only a few sensing lines to cover the whole length of a cord. The same topology allows us to embed fiber optic strands to integrate co-located visual feedback. We provide an overview of the enabling braiding techniques, design considerations, and approaches to gesture detection. These allow us to derive a set of interaction techniques, which we demonstrate with different form factors and capabilities. Our applications illustrate how I/O Braid can invisibly augment everyday objects, such as touch-sensitive headphones and interactive drawstrings on garments, while enabling discoverability and feedback through embedded light sources.

Orbits combines a visual display and an eye motion sensor to allow a user to select between options by tracking a cursor with the eyes as the cursor travels in a circular path around each option. Using an off-the-shelf Jins MEME pair of eyeglasses, we present a pilot study that suggests that the eye movement required for Orbits can be sensed using three electrodes: one in the nose bridge and one in each nose pad. For forced choice binary selection, we achieve a 2.6 bits per second (bps) input rate at 250ms per input. We also inntroduce Head Orbits, where the user fixates the eyes on a target and moves the head in synchrony with the orbiting target. Measuring only the relative movement of the eyes in relation to the head, this method achieves a maximum rate of 2.0 bps at 500ms per input. Finally, we combine the two techniques together with a gyro to create an interface with a maximum input rate of 5.0 bps.

The touchscreen has been the dominant input surface for smartphones and smartwatches. However, its small size compared to a phone limits the richness of the input gestures that can be supported. We present TapSkin, an interaction technique that recognizes up to 11 distinct tap gestures on the skin around the watch using only the inertial sensors and microphone on a commodity smartwatch. An evaluation with 12 participants shows our system can provide classification accuracies from 90.69% to 97.32% in three gesture families -- number pad, d-pad, and corner taps. We discuss the opportunities and remaining challenges for widespread use of this technique to increase input richness on a smartwatch without requiring further on-body instrumentation.

We explored symbolic canine-human communication for working dogs through the use of canine head gestures. We identified a set of seven criteria for selecting head gestures and identified the first four deserving further experimentation. We devised computationally inexpensive mechanisms to prototype the live system from a motion sensor on the dog's collar. Each detected gesture is paired with a predetermined message that is voiced to the humans by a smart phone. We examined the system and proposed gestures in two experiments, one indoors and one outdoors. Experiment A examined both gesture detection accuracy and a dog's ability to perform the gestures using a predetermined routine of cues. Experiment B examined the accuracy of this system on two outdoor working-dog scenarios. The detection mechanism we presented is sufficient to point to improvements into system design and provide valuable insights into which gestures fulfill the seven minimum criteria.

Passive Haptic Learning (PHL) enables users to acquire motor skills by receiving tactile stimulation while no perceived attention is given to learning. Initial work used gloves with embedded vibration motors to passively teach users how to play simple, one-handed, one-note-at-a-time piano melodies. In an effort to create a practical system for learning full piano pieces, we have developed a method of passively teaching two-handed chorded skills, initially focusing on Braille typing. Here, we extend this effort to piano and show that passive stimulation is more effective at teaching piano pieces when presented on both hands simultaneously as opposed to training the left hand and then the right, as is common in many active teaching methods. We also demonstrate that accompanying audio is not needed for passive learning of piano melodies, which allows mobile PHL gloves to be used in more everyday situations.

We explore using the Outer Ear Interface (OEI) to recognize eating activities. OEI contains a 3D gyroscope and a set of proximity sensors encapsulated in an off-the-shelf earpiece to monitor jaw movement by measuring ear canal deformation. In a laboratory setting with 20 participants, OEI could distinguish eating from other activities, such as walking, talking, and silently reading, with over 90% accuracy (user independent). In a second study, six subjects wore the system for 6 hours each while performing their normal daily activities. OEI correctly classified five minute segments of time as eating or non-eating with 93% accuracy (user dependent).

We present the results of an 8-week pilot study with 55 dogs investigating whether using quantimetric monitors and a companion smartphone application can reduce returns and increase the perceived strength of bonds between newly adopted dogs from the Humane Society of Silicon Valley and their adopters. Through this pilot study, we developed guidelines for future research and discovered promising results indicating that providing dog quantimetric data to adopters through the use of a smartphone application could yield reduced rates of re-relinquishment. Additionally, respondents indicated that they felt using the smartphone application helped them to better meet the activity needs of their dog and increased the bond between themselves and their newly adopted dog.

The study of dolphin cognition involves intensive research of animal vocalizations. Marine mammalogists commonly study a specific sound type known as the whistle found in dolphin communication. However, one of the main problems arises from noisy underwater environments. Often waves and splash noises will partially distort the whistle making analysis or extraction difficult. Another problem is discovering fundamental units that allow research of the composition of whistles. We propose a method for whistle extraction from noisy underwater recordings using a probabilistic approach. Furthermore, we investigate discovery algorithms for fundamental units using a mixture of hidden Markov models. We evaluate our findings with a marine mammalogist on data collected in the field. Furthermore, we have evidence that our algorithms enable researchers to form hypotheses about the composition of whistles.

Interacting with mobile technology while in-motion has become a daily activity for many of us. Common sense leads one to believe that texting with a mini-qwerty keyboard while mobile can be dangerous since users are distracted and not paying attention to the environment. Previous studies have found that mobility negatively impacts text entry performance for novice participants typing on virtual keyboards on touch screen mobile phones. We investigate the impact of mobility on expert users' ability to quickly and accurately input text on mobile phones equipped with fixed-key mini-qwerty keyboards. In total, 36 participants completed 600 minutes of typing on mini-qwerty keyboards (300 minutes training up to expertise) in three mobility conditions (seated, standing, and walking) generating almost 4,000,000 characters across all conditions. Surprisingly, we found that walking has a significant impact on expert typing speeds but does not significantly impact expert accuracy rates.

Computer-mediated interaction for working dogs is an important new domain for interaction research. In domestic settings, touchscreens could provide a way for dogs to communicate critical information to humans. In this paper we explore how a dog might interact with a touchscreen interface. We observe dogs' touchscreen interactions and record difficulties against what is expected of humans' touchscreen interactions. We also solve hardware issues through screen adaptations and projection styles to make a touchscreen usable for a canine's nose touch interactions. We also compare our canine touch data to humans' touch data on the same system. Our goal is to understand the affordances needed to make touchscreen interfaces usable for canines and help the future design of touchscreen interfaces for assistive dogs in the home.

We present our findings from the Electronic Textile Interface Swatch Book Workshops. The workshops were designed as the first in a series of collaborative design experiences that introduce small groups of faculty/students teams from particular design disciplines to the concept of electronic textile interfaces (ETIs) through the use of a textile interface “swatch book” with the support of technician/facilitators. The work here focuses on the experience of the working relationship between the designer participants and the more technologically oriented facilitators, rather than on how much the participants learned about technology. The contribution of this work is a an exploration into understanding how through the use of technology we can bridge the gap between the distant discipline expertise needed to work on projects like ETIs.

Language immersion from birth is crucial to a child's language development. However, language immersion can be particularly challenging for hearing parents of deaf children to provide as they may have to overcome many difficulties while learning American Sign Language (ASL). We are in the process of creating a mobile application to help hearing parents learn ASL. To this end, we have interviewed members of our target population to gain understanding of their motivations and needs when learning sign language. We found that the most common motivation for parents learning ASL is better communication with their children. Parents are most interested in acquiring more fluent sign language skills through learning to read stories to their children.

False positives are a common problem for interfaces that rely on gesture recognition. Often a gesture can seem fine in development but is found to trigger accidentally during an initial deployment of the interface, restarting development and increasing expense. In this work we introduce MAGIC 2.0, a technique for false positive prediction and prevention that can be used interactively during the interface design process. To ground our research, we implement MAGIC 2.0 as a web service and develop gesture interfaces using sensors on common Android mobile phone platforms. We use iSAX (indexable Symbolic Aggregate approXimation) to enable interactive searching (;1,500,000 sec) of everyday user movements on a standard workstation to determine if a candidate gesture will trigger accidentally during use of an interface. We perform a user-independent study that suggests that the number of matches to this Everyday Gesture Library (EGL) database is indeed predictive of a candidate gesture's suitability. We compare iSAX to hidden Markov models (HMMs) and nearest neighbor with respect to accuracy and speed for the EGL search. Using iSAX on the EGL, we also develop a “garbage” class and show that including this class in recognition reduces errors.

The CopyCat game is an interactive educational adventure game to help deaf children improve their language and memory abilities. As part of the CopyCat project, several computer-assisted language learning games have been designed, one of the games “Alien” is shown in Figure 1. Each game entails some sort of quest by the hero to collect items in order to solve a problem. In each quest, the child interacts with the hero (Iris the white cat) via American Sign Language (ASL) to warn her of a villain or identify where a hidden object is located. The child may view the tutor repeatedly if they so choose (see Figure 1). After the child talks to the hero, the child's signing is classified as correct or incorrect. If the child's signing is incorrect, a question mark appears above the hero's head to simulate misunderstanding by the hero, and the child must try again to communicate accurately. If the child's sign is correct, the hero, with the wave of a paw, “poofs” the villain, turning it into an innocuous item, and the hero continues on the quest.

Order picking is the process of collecting items from an assortment in inventory. It represents one of the main activities performed in warehouses and accounts for about 60% of the total operational costs of a warehouse. In previous work, we demonstrated the advantages of a head-mounted display (HMD) based picking chart over a traditional text-based pick list, a paper-based graphical pick chart, and a mobile pick-by-voice system. Here we perform two user studies that suggest that adding color cues and context sensing via a laser rangefinder improves picking accuracy with the HMD system. We also examine other variants of the pick chart, such as adding symbols, textual identifiers, images, and descriptions and their effect on accuracy, speed, and subjective usability.

We investigate the potential of the Kinect depth-mapping camera for sign language recognition and verification for educational games for deaf children. We compare a prototype Kinect-based system to our current CopyCat system which uses colored gloves and embedded accelerometers to track children's hand movements. If successful, a Kinect-based approach could improve interactivity, user comfort, system robustness, system sustainability, cost, and ease of deployment. We collected a total of 1000 American Sign Language (ASL) phrases across both systems. On adult data, the Kinect system resulted in 51.5% and 76.12% sentence verification rates when the users were seated and standing respectively. These rates are comparable to the 74.82% verification rate when using the current(seated) CopyCat system. While the Kinect computer vision system requires more tuning for seated use, the results suggest that the Kinect may be a viable option for sign verification.

Language immersion from birth is crucial to a child's language development. However, language immersion can be particularly challenging for hearing parents of deaf children to provide as they may have to overcome many difficulties while learning sign language. We intend to create a mobile device-based system to help hearing parents learn sign language. The first step is to understand what level of detail (i.e., resolution) is necessary for novice signers to learn from video of signs. In this paper we present the results of a study designed to evaluate the ability of novices learning sign language to ascertain the details of a particular sign based on video presented on a mobile device. Four conditions were presented. Three conditions involve manipulation of video resolution (low, medium, and high). The fourth condition employs insets showing the sign handshapes along with the high resolution video. Subjects were tested on their ability to emulate the given sign over 80 signs commonly used between parents and their young children. Although participants noticed a reduction in quality in the low resolution condition, there was no significant effect of condition on ability to generate the sign. Sign difficulty had a significant correlation with ability to correctly reproduce the sign. Although the inset handshape condition did not improve the participants' ability to emulate the signs correctly, participant feedback provided insight into situations where insets would be more useful, as well as further suggestions to improve video intelligibility. Participants were able to reproduce even the most complex signs tested with relatively high accuracy.

Devices capable of gestural interaction through motion sensing are increasingly becoming available to consumers; however, motion gesture control has yet to appear outside of game consoles. Interaction designers are frequently not expert in pattern recognition, which may be one reason for this lack of availability. Another issue is how to effectively test gestures to ensure that they are not unintentionally activated by a user's normal movements during everyday usage. We present MAGIC, a gesture design tool that addresses both of these issues, and detail the results of an evaluation.

Mobile Music Touch (MMT) helps teach users to play piano melodies while they perform other tasks. MMT is a lightweight, wireless haptic music instruction system consisting of fingerless gloves and a mobile Bluetooth enabled computing device, such as a mobile phone. Passages to be learned are loaded into the mobile phone and are played repeatedly while the user performs other tasks. As each note of the music plays, vibrators on each finger in the gloves activate, indicating which finger is used to play each note. We present two studies on the efficacy of MMT. The first measures 16 subjects' ability to play a passage after using MMT for 30 minutes while performing a reading comprehension test. The MMT system was significantly more effective than a control condition where the passage was played repeatedly but the subjects' fingers were not vibrated. The second study compares the amount of time required for 10 subjects to replay short, randomly generated passages using passive training versus active training. Participants with no piano experience could repeat the passages after passive training while subjects with piano experience often could not.

We present two experiments to evaluate wrist-worn wearable tactile displays (WTDs) that provide easy to perceive alerts for on-the-go users. The first experiment (2304 trials, 12 participants) focuses on the perception sensitivity of tactile patterns and reveals that people discriminate our 24 tactile patterns with up to 99% accuracy after 40 minutes of training. Among the four parameters (intensity, starting point, temporal pattern, and direction) that vary in the 24 patterns, intensity is the most difficult parameter to distinguish and temporal pattern is the easiest. The second experiment (9900 trials, 15 participants) focuses on dual task performance, exploring users' abilities to perceive three incoming alerts from two mobile devices (WTD and mobile phone) with and without visual distraction. The second experiment reveals that, when visually distracted, users' reactions to incoming alerts become slower for the mobile phone but not for the WTD.

Evaluations of task guidance systems often focus on evaluations of new technologies rather than comparing the nuances of interaction across the various systems. One common domain for task guidance systems is warehouse order picking. We present a method involving an easily reproducible ecologically motivated order picking environment for quantitative user studies designed to reveal differences in interactions. Using this environment, we perform a 12 participant within-subjects experiment demonstrating the advantages of a head-mounted display based picking chart over a traditional text-based pick list, a paper-based graphical pick chart, and a mobile pick-by-voice system. The test environment proved sufficiently sensitive, showing statistically significant results along several metrics with the head-mounted display system performing the best. We also provide a detailed analysis of the strategies adopted by our participants.

International and intercultural collaborations provide a unique opportunity to explore cultural differences in the usage and appropriation of a technology. Mobile photo capture and sharing has been growing in popularity in the Western world but nowhere has the practice been as eagerly adopted as in South Korea. In this paper we present an evaluation of a mobile-collocated photo sharing technology probe designed to determine the ways in which photo capture and sharing can effect and enhance face-to-face interaction for pre-existing social groups. We explore the interaction of culture and automatic, real-time photo capture and sharing on groups of friends engaging in a walking tour. We assemble a multicultural research team to better understand our observations and isolate cultural and technological artifacts. We relate our findings to prior work in the area to show that culture can have as much, if not more, impact on group usage of a technology than the technical capabilities of a system.

Classification of complex motor activities from brain imaging is relatively new in the fields of neuroscience and brain-computer interfaces (BCIs). We report sign language classification results for a set of three contrasting pairs of signs. Executed sign accuracy was 93.3%, and imagined sign accuracy was 76.7%. For a full multiclass problem, we used a decision directed acyclic graph of pairwise support vector machines, resulting in 63.3% accuracy for executed sign and 31.4% accuracy for imagined sign. Pairwise comparison of phrases composed of these signs yielded a mean accuracy of 73.4%. These results suggest the possibility of BCIs based on sign language.

We perform real-time American Sign Language (ASL) phrase verification for an educational game, CopyCat, which is designed to improve deaf children's signing skills. Taking advantage of context information in the game we verify a phrase, using Hidden Markov Models (HMMs), by applying a rejection threshold on the probability of the observed sequence for each sign in the phrase. We tested this approach using 1204 signed phrase samples from 11 deaf children playing the game during the phase two deployment of CopyCat. The CopyCat data set is particularly challenging because sign samples are collected during live game play and contain many variations in signing and disfluencies. We achieved a phrase verification accuracy of 83% compared to 90% real-time performance by a sign linguist. We report on the techniques required to reach this level of performance.

The natural language for most deaf signers in the United States is American Sign Language (ASL). ASL has internal structure like spoken languages, and ASL linguists have introduced several phonemic models. The study of ASL phonemes is not only interesting to linguists, but also useful for scalability in recognition by machines. Since machine perception is different than human perception, this paper learns the basic units for ASL directly from data. Comparing with previous studies, our approach computes a set of data-driven units (fenemes) discriminatively from the results of segmental feature selection. The learning iterates the following two steps: first apply discriminative feature selection segmentally to the signs, and then tie the most similar temporal segments to re-train. Intuitively, the sign parts indistinguishable to machines are merged to form basic units, which we call ASL fenemes. Experiments on publicly available ASL recognition data show that the extracted data-driven fenemes are meaningful, and recognition using those fenemes achieves improved accuracy at reduced model complexity.

When designing a text entry system for mobile phone keypads, a designer needs to overcome the ambiguity that arises from mapping the 26 letters of the roman alphabet to only 12 keys (0--9, *, #). In this paper, we present a novel two-thumb chording system for text entry on a standard 12-key mobile phone keypad and introduce a performance model based on Fitts' Law for an expert user. The model provides a behavioral description of the user and predicts a text entry rate of 55.02 wpm.

In this paper, we describe the design considerations and implementation of the Child'sPlay system, a technology for supporting the automatic recording, recognition, and quantification of a child's object play behaviors for retrospective analysis. Our prototype system consists of six varieties of toys augmented with wireless sensing capabilities and a mobile computing platform which uses statistical pattern recognition techniques to automatically classify sensed play behaviors. This paper discusses our choices in toy design both in form factor as well as sensing capabilities. In addition, we also describe the play activities the system supports and provide an overview of our initial recognition algorithms.

In this paper we present the results of a study designed to evaluate the computer-based methods of learning American Sign Language (ASL). We describe a method including an initial instruction session along with receptive and generative language tests which were administered after a week-long retention interval. We show a strong correlations (r=.62, r=.57) between the initial session's instruction and the receptive and generative levels of vocabulary signing. Based on the results of our experiment, we establish a baseline for further exploration of ASL vocabulary acquisition and identify further paths for language based instruction.

Seeking to enable direct and equal access for the Deaf to emergency call centers, we analyze the current state of the emergency phone system in the United States and elsewhere in the world. Leveraging teletypewriter (TTY) technology mandated by the Americans with Disabilities Act of 1990 to be installed in all emergency call centers in the United States, we developed software that emulates a TTY on a smart phone. We present an Instant Messaging style interface for mobile phones that uses the existing emergency infrastructure and allows Deaf users to communicate directly with emergency operators.

We investigate the effect of placement and user mobility on the time required to access an on-body interface. In our study, a wrist-mounted system was significantly faster to access than a device stored in the pocket or mounted on the hip. In the latter two conditions, 78% of the time it took to access the device was spent retrieving the device from its holder. As mobile devices are beginning to include peripherals (for example, Bluetooth headsets and watches connected to a mobile phone stored in the pocket), these results may help guide interface designers with respect to distributing functions across the body between peripherals.

By analyzing features of users' typing, Automatic Whiteout++ detects and corrects up to 32.37% of the errors made by typists while using a mini-QWERTY (RIM Blackberry style) keyboard. The system targets "off-by-one" errors where the user accidentally presses a key adjacent to the one intended. Using a database of typing from longitudinal tests on two different keyboards in a variety of contexts, we show that the system generalizes well across users, model of keyboard, user expertise, and keyboard visibility conditions. Since a goal of Automatic Whiteout++ is to embed it in the firmware of mini-QWERTY keyboards, it does not rely on a dictionary. This feature enables the system to correct errors mid-word instead of applying a correction after the word has been typed. Though we do not use a dictionary, we do examine the effect of varying levels of language context in the system's ability to detect and correct erroneous keypresses.

We address the feature selection problem for hidden Markov models (HMMs) in sequence classification. Temporal correlation in sequences often causes difficulty in applying feature selection tech niques. Inspired by segmental k-means segmentation (SKS) [B. Juang and L. Rabiner, 1990], we propose Segmentally Boosted HMMs (SBHMMs), where the state-optimized features are constructed in a segmental and discriminative manner. The contributions are twofold. First, we introduce a novel feature selection algorithm, where the temporal dynamics are decoupled from the static learning procedure by assuming that the sequential data are piecewise independent and identically distributed. Second, we show that the SBHMM consistently improves traditional HMM recognition in various domains. The reduction of error compared to traditional HMMs ranges from 17% to 70% in American Sign Language recognition, human gait identification, lip reading, and speech recognition.

The wristwatch is a device that is quick to access, but is currently under-utilized as a platform for interaction. We investigate interaction on a circular touchscreen wristwatch, empirically determining the error rate for variously-sized buttons placed around the rim. We consider three types of inter-target movements, and derive a mathematical model for error rate given a movement type and angular and radial button widths.

MacKenzie and Soukoreff have previously introduced a Fitts' Law-based performance model of expert two-thumb text entry on mini-QWERTY keyboards [4]. In this work we validate the original model using results from a longitudinal study of mini-QWERTY keyboards, and update the model to account for observed inter-key time data.

Discovering recurring patterns in time series data is a fundamental problem for temporal data mining. This paper addresses the problem of locating subdimensional motifs in real-valued, multivariate time series, which requires the simultaneous discovery of sets of recurring patterns along with the corresponding relevant dimensions. While many approaches to motif discovery have been developed, most are restricted to categorical data, univariate time series, or multivariate data in which the temporal patterns span all of the dimensions. In this paper, we present an expected linear-time algorithm that addresses a generalization of multivariate pattern discovery in which each motif may span only a subset of the dimensions. To validate our algorithm, we discuss its theoretical properties and empirically evaluate it using several data sets including synthetic data and motion capture data collected by an on-body iner- tial sensor.

CopyCat is an American Sign Language (ASL) game, which uses gesture recognition technology to help young deaf children practice ASL skills. We describe a brief history of the game, an overview of recent user studies, and the results of recent work on the problem of continuous, user-independent sign language recognition in classroom settings. Our database of signing samples was collected from user studies of deaf children playing aWizard of Oz version of the game at the Atlanta Area School for the Deaf (AASD). Our data set is characterized by disfluencies inherent in continuous signing, varied user characteristics including clothing and skin tones, and illumination changes in the classroom. The dataset consisted of 541 phrase samples and 1,959 individual sign samples of five children signing game phrases from a 22 word vocabulary. Our recognition approach uses color histogram adaptation for robust hand segmentation and tracking. The children wear small colored gloves with wireless accelerometers mounted on the back of their wrists. The hand shape information is combined with accelerometer data and used to train hidden Markov models for recognition. We evaluated our approach by using leave-one-out validation; this technique iterates through each child, training on data from four children and testing on the remaining child's data. We achieved average word accuracies per child ranging from 91.75% to 73.73% for the user-independent models.

In this paper we present a 20-participant controlled experiment to evaluate and compare a head-down visual display and a synthesized speech audio display for comprehending text while mobile. Participants completed reading comprehension trials while walking a path and sitting. We examine overall performance and perceived workload for four conditions: audio-walking, audio-sitting, visual-walking, and visual-sitting. Results suggest audio is an acceptable modality for mobile comprehension of text. Participants' comprehension scores for the audio-walking condition were comparable to the scores for the visual-walking condition. More importantly, participants saw improvements in their ability to navigate the environment when using the audio display.

The HandyKey Twiddler™ is a one-handed chording mobile keyboard that employs a 3 × 4 button design, similar to that of a standard mobile telephone. We present a longitudinal study of novice users' learning rates on the Twiddler. Ten participants typed for 20 sessions using 2 different text entry methods. Each session was composed of 20 min of typing with multitap and 20 min of one-handed chording on the Twiddler. We found that users initially had a faster average typing rate with multitap; however, after 4 sessions the difference became negligible, and by the 8th session participants typed faster with chording on the Twiddler. Five participants continued our study and achieved an average rate of 47 words per minute (wpm) after approximately 25 hr of practice in varying conditions. One participant achieved an average rate of 67 wpm, equivalent to the typing rate of the 2nd author, who has been a Twiddler user for 10 years.We analyze the effects of learning on various aspects of chording, provide evidence that lack of visual feedback does not hinder expert typing speed, and examine the potential use of multicharacter chords (MCCs) to increase text entry speed. Finally, we explore improving novice user's experience with the Twiddler through the use of a chording tutorial.

In order to provide relevant information to mobile users, such as workers engaging in the manual tasks of maintenance and assembly, a wearable computer requires information about the user's specific activities. This work focuses on the recognition of activities that are characterized by a hand motion and an accompanying sound. Suitable activities can be found in assembly and maintenance work. Here, we provide an initial exploration into the problem domain of continuous activity recognition using on-body sensing. We use a mock "wood workshop" assembly task to ground our investigation. We describe a method for the continuous recognition of activities (sawing, hammering, filing, drilling, grinding, sanding, opening a drawer, tightening a vise, and turning a screwdriver) using microphones and three-axis accelerometers mounted at two positions on the user's arms. Potentially "interesting" activities are segmented from continuous streams of data using an analysis of the sound intensity detected at the two different locations. Activity classification is then performed on these detected segments using linear discriminant analysis (LDA) on the sound channel and hidden Markov models (HMMs) on the acceleration data. Four different methods at classifier fusion are compared for improving these classifications. Using user-dependent training, we obtain continuous average recall and precision rates (for positive activities) of 78 percent and 74 percent, respectively. Using user-independent training (leave-one-out across five users), we obtain recall rates of 66 percent and precision rates of 63 percent. In isolation, these activities were recognized with accuracies of 98 percent, 87 percent, and 95 percent for the user-dependent, user-independent, and user-adapted cases, respectively.

We present a design for an interactive American Sign Language game geared for language development for deaf children. In addition to work on game design, we show how Wizard of Oz techniques can be used to facilitate our work on ASL recognition. We report on two Wizard of Oz studies which demonstrate our technique and maximize our iterative design process. We also detail specific implications to the design raised from working with deaf children and possible solutions.

We describe our initial efforts to learn high-level human behaviors from low-level gestures observed using on-body sensors. Such an activity discovery system could be used to index captured journals of a person's life automatically. In a medical context, an annotated journal could assist therapists in helping to describe and treat symptoms characteristic to behavioral syndromes such as autism. We review our current work on user-independent activity recognition from continuous data where we identify "interesting" user gestures through a combination of acceleration and audio sensors placed on the user's wrists and elbows. We examine an algorithm that can take advantage of such a sensor framework to automatically discover and label recurring behaviors, and we suggest future work where correlations of these low-level gestures may indicate higher-level activities

We present a Wizard of Oz evaluation of dual--purpose speech, a technique designed to provide support during a face--to--face conversation by leveraging a user's conversational speech for input. With a dual--purpose speech interaction, the user's speech is meaningful in the context of a human--to--human conversation while providing useful input to a computer. For our experiment, we evaluate the ability to schedule appointments with our calendaring application, the Calendar Navigator Agent. We examine the relative difference between using speech for input compared to traditional pen input on a PDA. We found that speech is more direct and our participants can use their conversational speech for computer input. In doing so, we reduce the manual input needed to operate a PDA while engaged in a calendaring conversation.

An experienced user of the Twiddler, a one--handed chording keyboard, averages speeds of 60 words per minute with letter--by--letter typing of standard test phrases. This fast typing rate coupled with the Twiddler's 3x4 button design, similar to that of a standard mobile telephone, makes it a potential alternative to multi--tap for text entry on mobile phones. Despite this similarity, there is very little data on the Twiddler's performance and learnability. We present a longitudinal study of novice users' learning rates on the Twiddler. Ten participants typed for 20 sessions using two different methods. Each session is composed of 20 minutes of typing with multi--tap and 20 minutes of one--handed chording on the Twiddler. We found that users initially have a faster average typing rate with multi--tap; however, after four sessions the difference becomes negligible, and by the eighth session participants type faster with chording on the Twiddler. Furthermore, after 20 sessions typing rates for the Twiddler are still increasing.

In this paper, we explore the concept of dual-purpose speech: speech that is socially appropriate in the context of a human-to-human conversation which also provides meaningful input to a computer. We motivate the use of dual-purpose speech and explore issues of privacy and technological challenges related to mobile speech recognition. We present three applications that utilize dual-purpose speech to assist a user in conversational tasks: the Calendar Navigator Agent, DialogTabs, and Speech Courier. The Calendar Navigator Agent navigates a user's calendar based on socially appropriate speech used while scheduling appointments. DialogTabs allows a user to postpone cognitive processing of conversational material by proving short-term capture of transient information. Finally, Speech Courier allows asynchronous delivery of relevant conversational information to a third party.

Video-based recognition and prediction of a temporally extended activity can benefit from a detailed description of high-level expectations about the activity. Stochastic grammars allow for an efficient representation of such expectations and are well-suited for the specification of temporally well-ordered activities. In this paper, we extend stochastic grammars by adding event parameters, state checks, and sensitivity to an internal scene model. We present an implemented system that uses human-specified grammars to recognize a person performing the Towers of Hanoi task from a video sequence by analyzing object interaction events. Experimental results from several videos show robust recognition of the full task and its constituent sub-tasks even though no appearance models of the objects in the video are provided. These experiments include videos of the task performed with different shaped objects and with distracting and extraneous interactions.

Gesture recognition is becoming a more common interaction tool in the fields of ubiquitous and wearable computing. Designing a system to perform gesture recognition, however, can be a cumbersome task. Hidden Markov models (HMMs), a pattern recognition technique commonly used in speech recognition, can be used for recognizing certain classes of gestures. Existing HMM toolkits for speech recognition can be adapted to perform gesture recognition, but doing so requires significant knowledge of the speech recognition literature and its relation to gesture recognition. This paper introduces the Georgia Tech Gesture Toolkit GT2k which leverages Cambridge University's speech recognition toolkit, HTK, to provide tools that support gesture recognition research. GT2k provides capabilities for training models and allows for both real--time and off-line recognition. This paper presents four ongoing projects that utilize the toolkit in a variety of domains.

Novel speech and/or gesture interfaces are candidates for use in future mobile or ubiquitous applications. This paper describes an evaluation of various interfaces for visual navigation of a whole Earth 3D terrain model. A mouse driven interface, a speech interface, a gesture interface, and a multimodal speech and gesture interface were used to navigate to targets placed at various points on the Earth. This study measured each participant's recall of target identity, order, and location as a measure of cognitive load. Timing information as well as a variety of subjective measures including discomfort and user preference were taken. While the familiar and mature mouse interface scored best by most measures, the speech interface also performed well. The gesture and multimodal interface suffered from weaknesses in the gesture modality. Weaknesses in the speech and multimodal modalities are identified and areas for improvement are discussed.

Computer gaming offers a unique test-bed and market for advanced concepts in computer science, such as Human Computer Interaction (HCI), computer-supported collaborative work (CSCW), intelligent agents, graphics, and sensing technology. In addition, computer gaming is especially well-suited for explorations in the relatively young fields of wearable computing and augmented reality (AR). This paper presents a developing multi-player augmented reality game, patterned as a cross between a martial arts fighting game and an agent controller, as implemented using the Wearable Augmented Reality for Personal, Intelligent, and Networked Gaming (WARPING) system. Through interactions based on gesture, voice, and head movement input and audio and graphical output, the WARPING system demonstrates how computer vision techniques can be exploited for advanced, intelligent interfaces.

The Perceptive Workbench enables a spontaneous, natural and unimpeded interface between the physical and virtual worlds. It uses vision-based methods for interaction that eliminate the need for wired input devices and wired tracking. Objects are recognized and tracked when placed on the display surface. Through the use of multiple light sources, the object's 3D shape can be captured and inserted into the virtual interface. This ability permits spontaneity since either preloaded objects or those objects selected on-the-spot by the user can become physical icons. Integrated into the same vision-based interface is the ability to identify 3D hand position, pointing direction and sweeping arm gestures. Such gestures can enhance selection, manipulation and navigation tasks. In this paper, the Perceptive Workbench is used for augmented reality gaming and terrain navigation applications, which demonstrate the utility and capability of the interface.

We present two real-time hidden Markov model-based systems for recognizing sentence-level continuous American sign language (ASL) using a single camera to track the user's unadorned hands. The first system observes the user from a desk mounted camera and achieves 92 percent word accuracy. The second system mounts the camera in a cap worn by the user and achieves 98 percent accuracy (97 percent with an unrestricted grammar). Both experiments use a 40-word lexicon.

We describe experiments with eigenfaces for recognition and interactive search in a large-scale face database. Accurate visual recognition is demonstrated using a database of O(10/sup 3/) faces. The problem of recognition under general viewing orientation is also examined. A view-based multiple-observer eigenspace technique is proposed for use in face recognition under variable pose. In addition, a modular eigenspace description technique is used which incorporates salient features such as the eyes, nose and mouth, in an eigenfeature layer. This modular representation yields higher recognition rates as well as a more robust framework for face recognition. An automatic feature extraction technique using feature eigentemplates is also demonstrated.>

A hidden Markov model (HMM) based continuous speech recognition system is applied to on-line cursive handwriting recognition. The base system is unmodified except for using handwriting feature vectors instead of speech. Due to inherent properties of HMMs, segmentation of the handwritten script sentences is unnecessary. A 1.1% word error rate is achieved for a 3050 word lexicon, 52 character, writer-dependent task and 3%-5% word error rates are obtained for six different writers in a 25,595 word lexicon, 86 character, writer-dependent task. Similarities and differences between the continuous speech and on-line cursive handwriting recognition tasks are explored; the handwriting database collected over the past year is described; and specific implementation details of the handwriting system are discussed.>

Interactive graphics systems that are driven by visual input are discussed. The underlying computer vision techniques and a theoretical formulation that addresses issues of accuracy, computational efficiency, and compensation for display latency are presented. Experimental results quantitatively compare the accuracy of the visual technique with traditional sensing. An extension to the basic technique to include structure recovery is discussed.>

Abstract To be convincing and natural, interactive graphics applications must correctly synchronize user motion with rendered graphics and sound output. We present a solution to the synchronization problem that is based on optimal estimation methods and fixed-lag dataflow techniques. A method for discovering and correcting prediction errors using a generalized likelihood approach is also presented. And finally, Music World, a simulated environment employing these ideas, is described.