Playing video games can enhance older adults’ well-being and social connections. However, most mainstream games rely on button-based controls that require fine motor skills, limiting accessibility. We present ElderPlay, a real-time game input translation system that enables older adults to play unmodified commercial games using intuitive, motion-based interaction. We first conducted a gesture elicitation study to derive user-defined gestures grounded in everyday experiences, which informed the design of a proof-of-concept system translating gestures into controller inputs. We then evaluated ElderPlay with two commercial Nintendo Switch games. Results show that gesture-based interaction improves enjoyment, perceived physical engagement, and performance. Rather than replacing controllers, our findings highlight the effectiveness of hybrid interaction, where gesture and controller inputs support different gameplay actions. We discuss implications for context-dependent and inclusive game interaction design.

Urban development reduces green space, disrupts wildlife, and increases noise, diminishing everyday exposure to natural soundscapes and weakening people’s connection to nearby nature. We explore restoring urban natural soundscapes with personal Audio AR to help people reacquaint with everyday nature, and observe associations with perceived nature connectedness, eco-awareness (noticing both existing and missing greenery), and reflection. We present GreenAR, a ubiquitous location-aware system that composes spatial, visually-congruent biophony (birds, insects) from OpenStreetMap greenery around the listener. We conducted three in-situ studies: (1) an error-resilient sound placement evaluation under urban GPS conditions (n=16); (2) a campus study comparing with/without restoration (day/night) (n=16); and (3) a week-long field deployment in five countries (n=12). Participants reported higher nature connectedness with restoration, heightened awareness of vegetation and urban wildlife, attention to green deficits along routes, and occasional micro‑actions. Notably, one participant described a reversal from opposing to supporting urban greening policies, highlighting the potential of everyday ambient soundscapes to shape reflection on urban nature.

Current co-op games keep collaboration virtual even when players are physically co-located in the same room, limiting embodied coordination in the shared space. We introduce MoveTogether, a novel physical co-op gameplay in which two players jointly operate a single, tracked prop, adding a shared physical communication channel on top of visual and audio cues. To explore the design space in mixed reality, we conducted a workshop with 10 professional designers, generating a physical co-op design space that encompasses prop and interaction design patterns, and how they relate to affordance and cooperative experience. In a within-subjects study of virtual vs. physical co-op experiences (n=16), we observed finer-grained task coordination, fewer collisions, and more strategy-focused communication. Players reported higher collaboration, sense of achievement, enjoyment, and overall preference for physical co-op. This work opens a new design space for co-located play and offers guidance for designing embodied co-op experiences.

Family co-play of video games is common and linked to family closeness and positive intergenerational interaction. Mixed-age co-play exposes controller-skill gaps that cause frustration and exclusion, yet most party games lack ability-inclusive assists. We explore ability-inclusive co-play for unmodified party and family games: We contribute (1) an open-source controller-skills benchmark; (2) the first lifespan study combining both broad age and deep skill coverage (ages 6–90; n=80; six skills), showing that gaps between children and older adults are skill-specific rather than uniform; (3) we present PartyAssist, a real-time, computer-vision-to-input system that detects on-screen state and injects micro controller-input assistance without altering game code; (4) in a feasibility study with 16 mixed-skill dyads (n=32; assisted player <10 or >50), assistance improved children’s survival time and success rate while remaining largely unnoticed and was viewed positively, while older adults detected assistance and reported mixed views. Interviews surfaced socially-considerate design nuances and implications to inform future designs to support ability-inclusive co-play across the lifespan.

Concept designers in the entertainment industry create highly detailed, often imaginary environments for movies, games, and TV shows. Their early ideation phase requires intensive research, brainstorming, visual exploration, and combination of various design elements to form cohesive designs. However, existing AI tools focus on image generation from user specifications, lacking support for the unique needs and complexity of concept designers' workflows. Through a formative study with 12 professional designers, we captured their workflows and identified key requirements for AI-assisted ideation tools. Leveraging these insights, we developed AIdeation to support early ideation by brainstorming design concepts with flexible searching and recombination of reference images. A user study with 16 professional designers showed that AIdeation significantly enhanced creativity, ideation efficiency, and satisfaction (all p<.01) compared to current tools and workflows. A field study with 4 studios for 1 week provided insights into AIdeation's benefits and limitations in real-world projects. After the completion of the field study, two studios, covering films, television, and games, have continued to use AIdeation in their commercial projects to date, further validating AIdeation's improvement in ideation quality and efficiency.

Environment designers in the entertainment industry create imaginative 2D and 3D scenes for games, films, and television, requiring both fine-grained control of specific details and consistent global coherence. Designers have increasingly integrated generative AI into their workflows, often relying on large language models (LLMs) to expand user prompts for text-to-image generation, then iteratively refining those prompts and applying inpainting. However, our formative study with 10 designers surfaced two key challenges: (1) the lengthy LLM-generated prompts make it difficult to understand and isolate the keywords that must be revised for specific visual elements; and (2) while inpainting supports localized edits, it can struggle with global consistency and correctness. Based on these insights, we present GenTune, an approach that enhances human--AI collaboration by clarifying how AI-generated prompts map to image content. Our GenTune system lets designers select any element in a generated image, trace it back to the corresponding prompt labels, and revise those labels to guide precise yet globally consistent image refinement. In a summative study with 20 designers, GenTune significantly improved prompt--image comprehension, refinement quality, and efficiency, and overall satisfaction (all $p < .01$) compared to current practice. A follow-up field study with two studios further demonstrated its effectiveness in real-world settings.

Research has shown that walking during meetings improves creativity, memory, attention, health, and happiness. While mobile technologies have freed users from having to be stationary during virtual meetings, mobile phones pose several usability challenges, such as reduced productivity due to small screens and safety concerns. In this paper, we present the first exploration of augmented reality (AR) interface design for virtual meetings in real-world walking conditions. We conducted design sessions in-situ with 16 user interface and AR designers using a 2x2 experimental design: meeting while walking in two levels of traffic conditions and with two types of meeting formats. Results show that the designed AR windows averaged 14.5 times in viewing size compared to the most popular smartphone screens. Also, Traffic Level significantly affected the size, opacity, and placement of windows, as well as the preference of anchoring modes, while Meeting Format significantly affected size and opacity. Furthermore, clustering analysis identified two groups of designs that can serve as initial reference designs for further customization and research.

Electrical Muscle Stimulation (EMS) has emerged as a key wearable haptic feedback technology capable of simulating a wide range of force feedback, such as the impact force of boxing punches, the weight of virtual objects, and the reaction force from pushing on a wall. To simulate these external forces, EMS stimulates the muscles that oppose (i.e. antagonistic to) the actual muscles that users activate, causing involuntary muscle contraction and haptic sensations that differ from real-world experiences. In this work, we propose Paired-EMS which simultaneously stimulates both the muscles that users activate and that prior EMS stimulates (i.e. antagonistic muscle pairs) to enhance the external force feedback experience. We first conducted a small formative study (n=8) to help design the stimulation intensity of muscle pairs, then conducted a user experience study to evaluate Paired-EMS vs. prior EMS approaches for both isometric and isotonic user actions. Study results (n=32) showed that Paired-EMS significantly improved realism, harmony, and entertainment (p<.05) with similar comfort (p>.36), and was overall preferred by 78% of participants (p<.01).

This paper explores the feasibility of deliberately designing VR motion that diverges from users’ physical movements to turn mundane, everyday transportation motion (e.g., metros, trains, and cars) into more entertaining VR motion experiences, in contrast to prior car-based VR approaches that synchronize VR motion to physical car movement exactly. To gain insight into users’ preferences for veering rate and veering direction for turning (left/right) and pitching (up/down) during the three phases of acceleration (accelerating, cruising, and decelerating), we conducted a formative, perceptual study (n=24) followed by a VR experience evaluation (n=18), all conducted on metro trains moving in a mundane, straight-line motion. Results showed that participants preferred relatively high veering rates, and preferred pitching upward during acceleration and downward during deceleration. Furthermore, while veering decreased comfort as expected, it significantly enhanced immersion (p<.01) and entertainment (p<.001) and the overall experience, with comfort being considered, was preferred by 89% of participants.

Interior design aims to create aesthetically pleasing and functional environments within an architectural space. For a simple room, the preliminary design exploration currently takes multiple meetings and days of work for interior designers to incorporate homeowners’ personal preferences through layout, furnishings, form, colors, and materials. We present RoomDreaming, a generative AI-based approach designed to facilitate preliminary interior design exploration. It empowers owners and designers to rapidly and efficiently iterate through a broad range of AI-generated, photo-realistic design alternatives, each uniquely tailored to fit actual space layouts and individual design preferences. We conducted a series of formative and summative studies with a total of 18 homeowners and 20 interior designers to help design, improve, and evaluate RoomDreaming. Owners reported that RoomDreaming effectively increased the breadth and depth of design exploration with higher efficiency and satisfaction. Designers reported that one hour of collaborative designing with RoomDreaming yielded results comparable to several days of traditional owner-designer meetings, plus days to weeks worth of designer work to develop and refine designs.

Real-world impact, such as hitting a tennis ball and a baseball, generates instantaneous, directional impact forces. However, current ungrounded force feedback technologies, such as air jets and propellers, can only generate directional impulses that are 10x-10,000x weaker. We present SpinShot, a flywheel-based device with a solenoid-actuated stopper capable of generating directional impulse of 22Nm in 1ms, which is more than 10x stronger than prior ungrounded directional technologies. Furthermore, we present a novel force design that reverses the flywheel immediately after the initial impact, to significantly increase the perceived magnitude. We conducted a series of two formative, perceptual studies (n=16, 18), followed by a summative user experience study (n=16) that compared SpinShot vs. moving mass (solenoid) and vs. air jets in a VR baseball hitting game. Results showed that SpinShot significantly improved realism, immersion, magnitude (p < .01) compared to both baselines, but significantly reduced comfort vs. air jets primarily due to the 2.9x device weight. Overall, SpinShot was preferred by 63-75% of the participants.

Mobile phones have enabled users to browse information in varying mobility contexts. For high-mobility settings such as walking, however, phones pose several usability challenges, particularly safety and limited screen sizes. While Augmented Reality (AR) has been proposed to address these issues, prior work has yet to investigate AR interface design in real-world walking conditions beyond text readability and notification design. This paper presents the first exploration of AR browsing interface design and extended usage while walking in the wild. We first conducted design sessions with 12 UI designers while walking in varied environments to design the window size, distance, opacity, anchor type, and placement for three categories of apps: text, video, and mixed content. Results show that traffic level significantly affects the designed window size, whereas content type significantly affects window size, distance, opacity, and vertical placement. To gain further insights from real-world usage, we conducted a multi-day observational study with 5 participants and observed that participants on average switched among window layouts every 3.3 minutes, for reasons such as safety and the level of extended visual attention.

First-Person View (FPV) drone is a recently developed category of drones designed for precision flying and for capturing exhilarating experiences that could not be captured before, such as navigating through tight indoor spaces and flying extremely close to subjects of interest. FPV viewing experiences, while exhilarating, typically have frequent rotations that can lead to visually induced discomfort. We present TurnAhead, which uses 3-DoF rotational haptic cues that correspond to camera rotations to improve the comfort, immersion, and enjoyment of FPV experiences. It uses headset-mounted air jets to provide ungrounded rotational forces and is the first device to support rotation around all 3 axes: yaw, pitch, and roll. We conducted a series of perception and formative studies to explore the design space of timing and intensity of haptic cues, followed by user experience evaluation, for a combined total of 44 participants (n=12, 8, 6, 18). Results showed that TurnAhead significantly improved overall comfort, immersion, and enjoyment, and was preferred by 89% of participants.

Impact events, which generate directional forces with extremely short impulse durations and large force magnitudes, are prevalent in both virtual reality (VR) games and real-world experiences. However, despite recent advancement in ungrounded force feedback technologies, such as air jet propulsion and propellers, these technologies remain 5-100x weaker and 10-500x slower compared to real-world impact events. For instance, they can only achieve 4N with a minimal duration of 50-500ms compared to the 20-400N forces generated within 1-5ms for baseball, ping-pong, drumming, and tennis. To overcome these limitations, we present AirCharge, a novel haptic device that accumulates air propulsion momentum to generate instantaneous, directional impact forces. By mounting compressed air jets on rotating swingarms, AirCharge can amplify impact force magnitude by more than 10x while matching real-world impulse duration of 3ms. To support high-frequency impacts, we explored and evaluated a series of device designs, culminating in a novel reciprocating dual-swingarm design that leverages a reversing bevel gearbox to eliminate gyro effects and to achieve impact feedback of up to 10Hz. User experience evaluation (n = 16) showed that AirCharge significantly enhanced realism and is preferred by participants compared to air jets without the charging mechanism.

We present DrivingVibe, which explores vibrotactile feedback designs around the head to enhance VR driving motion experiences. We propose two approaches that use a 360-degree vibrotactile headband: 1) mirroring and 2) 3D inertia-based. The mirroring approach extends the vibrotactile patterns of handheld controllers to actuate the entire headband uniformly. The 3D inertia-based approach uses the acceleration telemetry data that driving games/simulators export to motion platforms to generate directional vibration patterns, including: i) centrifugal forces, ii) horizontal acceleration/deceleration, and iii) vertical motion due to rough terrain. The two approaches are complementary as the mirroring approach supports all driving games because it does not require telemetry data, while the 3D inertia-based approach provides higher feedback fidelity for games that provide such data. We conducted a 24-person user experience evaluation in both passive passenger mode and active driving mode. Study results showed that both DrivingVibe designs significantly improved realism, immersion, and enjoyment (p<.01) with large effect sizes for the VR driving experiences. For overall preference, 88% (21/24) of participants preferred DrivingVibe, with a 2:1 preference for 3D inertia-based vs. mirroring designs (14 vs. 7 participants). For immersion and enjoyment, 96% (23/24) of participants preferred DrivingVibe, with nearly a 3:1 preference (17 vs. 6 participants) for the 3D inertia-based design.

We present AirRacket, perceptual modeling and design of ungrounded, directional force feedback for virtual racket sports. Using compressed air propulsion jets to provide directional impact forces, we iteratively designed for three popular sports that span a wide range of force magnitudes: ping-pong, badminton, and tennis. To address the limited force magnitude of ungrounded force feedback technologies, we conducted a perception study which discovered the novel illusion that users perceive larger impact force magnitudes with longer impact duration, by an average factor of 2.57x. Through a series of formative, perceptual, and user experience studies with a combined total of 72 unique participants, we explored several perceptual designs using force magnitude scaling and duration scaling methods to expand the dynamic range of perceived force magnitude. Our user experience evaluation showed that perceptual designs can significantly improve realism and preference vs. physics-based designs for ungrounded force feedback systems.

Teleportation, which instantly moves users from their current location to the target location, has become the most popular locomotion technique in VR games. It enables fast navigation with reduced VR sickness but results in significantly reduced immersion. We present HeadWind, a novel approach to improve the experience of teleportation by simulating the haptic sensation of air drag when rapidly moving through the air in real life. Specifically, HeadWind modulates bursts of compressed air to the face and uses multiple nozzles to provide directional cues. To design the wearable device and to model airflow speed and duration for teleportation, we conducted three formative studies and a design session. User experience evaluation with 24 participants showed that HeadWind significantly improved realism, immersion, and enjoyment of teleportation in VR (p<.01) with large effect sizes (r>0.5), and was preferred by 96% of participants.

Haptic feedback significantly enhances virtual experiences. However, supporting haptics currently requires modifying the codebase, making it impractical to add haptics to popular, high-quality experiences such as best selling games, which are typically closed-source. We present HapticSeer, a multi-channel, black-box, platform-agnostic approach to detecting game events for real-time haptic feedback. The approach is based on two key insights: 1) all games have 3 types of data streams: video, audio, and controller I/O, that can be analyzed in real-time to detect game events, and 2) a small number of user interface design patterns are reused across most games, so that event detectors can be reused effectively. We developed an open-source HapticSeer framework and implemented several real-time event detectors for commercial PC and VR games. We validated system correctness and real-time performance, and discuss feedback from several haptics developers that used the HapticSeer framework to integrate research and commercial haptic devices.

JetController is a novel haptic technology capable of supporting high-speed and persistent 3-DoF ungrounded force feedback. It uses high-speed pneumatic solenoid valves to modulate compressed air to achieve 20-50Hz of full impulses at 4.0-1.0N, and combines multiple air propulsion jets to generate 3-DoF force feedback. Compared to propeller-based approaches, JetController supports 10-30 times faster impulse frequency, and its handheld device is significantly lighter and more compact. JetController supports a wide range of haptic events in games and VR experiences, from firing automatic weapons in games like Halo (15Hz) to slicing fruits in Fruit Ninja (up to 45Hz). To evaluate JetController, we integrated our prototype with two popular VR games, Half-life: Alyx and Beat Saber, to support a variety of 3D interactions. Study results showed that JetController significantly improved realism, enjoyment, and overall experience compared to commercial vibrating controllers, and was preferred by most participants.

We present MotionRing, a vibrotactile headband that creates illusory tactile motion around the head by controlling the timing of a 1-D, 360° sparse array of vibration motors. Its unique ring shape enables symmetric and asymmetric haptic motion experiences, such as when users pass through a medium and when an object passes nearby in any direction. We first conducted a perception study to understand how factors such as vibration motor timing, spacing, duration, intensity, and head region affect the perception of apparent tactile motion. Results showed that illusory tactile motion around the head can be achieved with 12 and 16 vibration motors with angular speed between 0.5-4.9 revolutions per second. We developed a symmetric and an asymmetric tactile motion pattern to enhance the experience of teleportation in VR and dodging footballs, respectively. We conducted a user study to compare the experience of MotionRing vs. static vibration patterns and visual-only feedback. Results showed that illusory tactile motion significantly improved users’ perception of directionality and enjoyment of motion events, and was most preferred by users.

Virtual Reality (VR) sickness is common with symptoms such as headaches, nausea, and disorientation, and is a major barrier to using VR. We propose WalkingVibe, which applies unobtrusive vibrotactile feedback for VR walking experiences, and also reduces VR sickness and discomfort while improving realism. Feedback is delivered through two small vibration motors behind the ears at a frequency that strikes a balance in inducing vestibular response while minimizing annoyance. We conducted a 240-person study to explore how visual, audio, and various tactile feedback designs affect the locomotion experience of users walking passively in VR while seated statically in reality. Results showed timing and location for tactile feedback have significant effects on VR sickness and realism. With WalkingVibe, 2-sided step-synchronized design significantly reduces VR sickness and discomfort while significantly improving realism. Furthermore, its unobtrusiveness and ease of integration make WalkingVibe a practical approach for improving VR experiences with new and existing VR headsets.

We present Miniature Haptics, a new approach to providing realistic haptic experiences by applying miniaturized haptic feedback to hand-based, embodied avatars. By shrinking haptics to a much smaller scale, Miniature Haptics enables the exploration of new haptic experiences that are not practical to create at the full, human-body scale. Using Finger Walking in Place (FWIP) as an example avatar embodiment and control method, we first explored the feasibility of Miniature Haptics then conducted a human factors study to understand how people map their full-body skeletal model to their hands. To understand the user experience of Miniature Haptic, we developed a miniature football haptic display, and results from our user study show that Miniature Haptics significantly improved the realism and enjoyment of the experience and is preferred by users (p < 0.05). In addition, we present two miniature motion platforms supporting the haptic experiences of: 1) rapidly changing ground height for platform jumping games such as Super Mario Bros and 2) changing terrain slope. Overall, Miniature Haptics makes it possible to explore novel haptic experiences that have not been practical before.

Strength training improves overall health, well-being, physical appearance, and sports performance, with training programs specifying variables such as sets, repetitions, rest time, weight, and tempo. The repetitive nature of strength training, typically performed at fixed tempo, has made it challenging to develop entertaining exergames for common strength training exercises. We present StrengthGaming, which uses scaling and shuffling technique to provide tempo variations to strength training while preserving the training volume and training goals. It affords game designers more flexibility in designing strength training-based exergames. We developed a prototype game, inspired by FlappyBird, that uses a wearable orientation sensor to track the repetition tempo to control a flying character, using both fixed tempo design and dynamic tempo design. Results from our 24-person user study showed that dynamic tempo was significantly more entertaining than fixed tempo (p<0.01), and was preferred by participants.

Strength training improves overall health, well-being, physical appearance, and sports performance.There are four major factors that affect training efficacy in a training session: exercise type, number of repetitions, movement velocity, and workload. Prior research has used wearable sensors to detect exercise type, number of repetitions, and movement velocity while training. However, detecting workload remains constrained to instrumented exercise equipment, such as smart exercise machines or RFID-tagged free weights.This paper presents MuscleSense, an approach that estimates exercise workload by using wearable Surface Electromyography (sEMG) sensors and regression analysis. We evaluated the accuracy of several regression models and the effects of sensor placement through a 20-person user study. Results showed that MuscleSense achieved an accuracy of 0.68kg (root mean square error, RMSE) in sensing workload using both forearm and arm sensors and support vector regression (SVR).

We present HeadBlaster, a novel wearable technology that creates motion perception by applying ungrounded force to the head to stimulate the vestibular and proprioception sensory systems. Compared to motion platforms that tilt the body, HeadBlaster more closely approximates how lateral inertial and centrifugal forces are felt during real motion to provide more persistent motion perception. In addition, because HeadBlaster only actuates the head rather than the entire body, it eliminates the mechanical motion platforms that users must be constrained to, which improves user mobility and enables room-scale VR experiences. We designed a wearable HeadBlaster system with 6 air nozzles integrated into a VR headset, using compressed air jets to provide persistent, lateral propulsion forces. By controlling multiple air jets, it is able to create the perception of lateral acceleration in 360 degrees. We conducted a series of perception and human-factor studies to quantify the head movement, the persistence of perceived acceleration, and the minimal level of detectable forces. We then explored the user experience of HeadBlaster through two VR applications: a custom surfing game, and a commercial driving simulator together with a commercial motion platform. Study results showed that HeadBlaster provided significantly longer perceived duration of acceleration than motion platforms. It also significantly improved realism and immersion, and was preferred by users compared to using VR alone. In addition, it can be used in conjunction with motion platforms to further augment the user experience.

Modern touchscreen devices have recently introduced customizable touchscreen settings to improve accessibility for users with motor impairments. For example, iOS 10 introduced the following four Touch Accommodation settings: 1) Hold Duration, 2) Ignore Repeat, 3) Tap Assistance, and 4) Tap Assistance Gesture Delay. These four independent settings lead to a total of more than 1 million possible configurations, making it impractical to manually determine the optimal settings. We present PersonalTouch, which collects and analyzes touchscreen gestures performed by individual users, and recommends personalized, optimal touchscreen accessibility settings. Results from our user study show that PersonalTouch significantly improves touch input success rate for users with motor impairments (20.2%, N=12, p=.00054) and for users without motor impairments (1.28%, N=12, p=.032).

Augmented Reality (AR) provides real-time information by superimposing virtual information onto users' view of the real world. Our work is the first to explore how peripheral vision, instead of central vision, can be used to read text on AR and smart glasses. We present Peritext, a multiword reading interface using rapid serial visual presentation (RSVP). This enables users to observe the real world using central vision, while using peripheral vision to read virtual information. We first conducted a lab-based study to determine the effect of different text transformation by comparing reading efficiency among 3 capitalization schemes, 2 font faces, 2 text animation methods, and 3 different numbers of words for RSVP paradigm. We found that title case capitalization, sans-serif font and word-wise typewriter animation with multiword RSVP display resulted in better reading efficiency, which together formed our Peritext design. Another lab-based study followed, investigating the performance of the Peritext against control text, and the results showed significant better performance. Finally, we conducted a field study to collect user feedback while using Peritext in real-world walking scenarios, and all users reported a preference of 5° eccentricity over 8°.

Virtual Reality (VR) sickness occurs when exposure to a virtual environment causes symptoms that are similar to motion sickness, and has been one of the major user experience barriers of VR. To reduce VR sickness, prior work has explored dynamic field-of-view modification and galvanic vestibular stimulation (GVS) that recou-ples the visual and vestibular systems. We propose a new approach to reduce VR sickness, called PhantomLegs, that applies alternating haptic cues that are synchronized to users' footsteps in VR. Our prototype consists of two servos with padded swing arms, one set on each side of the head, that lightly taps the head as users walk in VR. We conducted a three-session, multi-day user study with 30 participants to evaluate its effects as users navigate through a VR environment while physically being seated. Results show that our approach significantly reduces VR sickness during the initial exposure while remaining comfortable to users.

Deaf and hard-of-hearing (DHH) individuals encounter difficulties when engaged in group conversations with hearing individuals, due to factors such as simultaneous utterances from multiple speakers and speakers whom may be potentially out of view. We interviewed and co-designed with eight DHH participants to address the following challenges: 1) associating utterances with speakers, 2) ordering utterances from different speakers, 3) displaying optimal content length, and 4) visualizing utterances from out-of-view speakers. We evaluated multiple designs for each of the four challenges through a user study with twelve DHH participants. Our study results showed that participants significantly preferred speechbubble visualizations over traditional captions. These design preferences guided our development of SpeechBubbles, a real-time speech recognition interface prototype on an augmented reality head-mounted display. From our evaluations, we further demonstrated that DHH participants preferred our prototype over traditional captions for group conversations.

Proper ergonomics improves productivity and reduces risks for injuries such as tendinosis, tension neck syndrome, and back injuries. Despite having ergonomics standards and guidelines for computer usage since the 1980s, injuries due to poor ergonomics remain widespread. We present ActiveErgo, the first active approach to improving ergonomics by combining sensing and actuation of motorized furniture. It provides automatic and personalized ergonomics of computer workspaces in accordance to the recommended ergonomics guidelines. Our prototype system uses a Microsoft Kinect sensor for skeletal sensing and monitoring to determine the ideal furniture positions for each user, then uses a combination of automatic adjustment and real-time feedback to adjust the computer monitor, desk, and chair positions. Results from our 12-person user study demonstrated that ActiveErgo significantly improves ergonomics compared to manual configuration in both speed and accuracy, and helps significantly more users to fully meet ergonomics guidelines.

Drones offer camera angles that are not possible with traditional cameras and are becoming increasingly popular for videography. However, flying a drone and controlling its camera simultaneously requires manipulating 5-6 degrees of freedom (DOF) that needs significant training. We present ARPilot, a direct-manipulation interface that lets users plan an aerial video by physically moving their mobile devices around a miniature 3D model of the scene, shown via Augmented Reality (AR). The mobile devices act as the viewfinder, making them intuitive to explore and frame the shots. We leveraged AR technology to explore three 6DOF video-shooting interfaces on mobile devices: AR keyframe, AR continuous, and AR hybrid, and compared against a traditional touch interface in a user study. The results show that AR hybrid is the most preferred by the participants and expends the least effort among all the techniques, while the users' feedback suggests that AR continuous empowers more creative shots. We discuss several distinct usage patterns and report insights for further design.

Augmented Reality (AR) provides real-time information by superimposing virtual information onto users' view of the real world. Our work is the first to explore how peripheral vision, instead of central vision, can be used to read text on AR and smart glasses. We present PeriTextAR, a multiword reading interface using rapid serial visual presentation (RSVP)[5]. This enables users to observe the real world using central vision, while using peripheral vision to read virtual information. We first conducted a lab-based study to determine the effect of different text transformation by comparing reading efficiency among 3 capitalization schemes, 2 font faces, 2 text animation methods, and 3 different numbers of words for RSVP paradigm. Another lab-based study followed, investigating the performance of the PeriTextAR against control text, and the results showed significant better performance.

Inequality of communication capabilities may cause an awkward experience in real life. %However, in terms of gaming, unequal communication mechanic can make games more challenging and interesting. However, in terms of gaming, this can be a challenging and interesting mechanic. Charades[1], where one player tries to express a word via body language while others try to guess it, is a good example of such cases. Different from most digital co-op games, which equal communication mechanics have been widely used, we experiment with this unequal communication mechanic. Thus, we have designed a digital co-op game called getGameName, in which 2 avatars, a human and a dog, communicate to solve a series of puzzles with unequal communication capabilities. Ultimately, we conducted an experiment to evaluate our game. Several interesting patterns were discovered from our observation. We then concluded some design suggestions from our collected data for future reference.

Electric current and voltage are fundamental to learning, understanding, and debugging circuits. Although both can be measured using tools such as multimeters and oscilloscopes, electric current is much more difficult to measure because users have to unplug parts of a circuit and then insert the measuring tools in serial. Furthermore, users need to restore the circuits back to its original state after measurements have been taken. In practice, this cumbersome process poses a formidable barrier to knowing how current flows throughout a circuit. We present CurrentViz, a system that can sense and visualize the electric current flowing through a circuit, which helps users quickly understand otherwise invisible circuit behavior. It supports fully automatic, ubiquitous, and real-time collection of amperage information of breadboarded circuits. It also supports visualization of the amperage data on a circuit schematic to provide an intuitive view into the current state of a circuit.

The rise of Maker communities and open-source electronic prototyping platforms have made electronic circuit projects increasingly popular around the world. Although there are software tools that support the debugging and sharing of circuits, they require users to manually create the virtual circuits in software, which can be time-consuming and error-prone. We present CircuitSense, a system that automatically recognizes the wires and electronic components placed on breadboards. It uses a combination of passive sensing and active probing to detect and generate the corresponding circuit representation in software in real-time. CircuitSense bridges the gap between the physical and virtual representations of circuits. It enables users to interactively construct and experiment with physical circuits while gaining the benefits of using software tools. It also dramatically simplifies the sharing of circuit designs with online communities.

Current virtual reality (VR) and augmented reality (AR) devices rely on handheld devices, hand gestures, head tracking, and voice input. This paper presents an initial exploration of using eye expressions as hands-free input modality for head-mounted AR/VR devices (HMDs). We consulted interaction designers and ophthalmologists and enumerated 12 eye expressions, and conducted a 12-person user study to better understand users' ability to perform them as well as preferences.

Augmented Reality (AR) provides real-time information by super-imposing virtual information onto users' view of the real world. Our work is the first to explore how peripheral vision, instead of central vision, can be used to read text on AR and smart glasses. We present PeriText+, a multiword reading interface using rapid serial visual presentation (RSVP). This enables users to observe the real world using central vision, while using peripheral vision to read text. We conducted a lab study to compare reading efficiency among 40 different conditions of text transformation. We also conducted a field study to evaluate the information transfer while using PeriText+ in a real-world walking scenario.

Force sensing has been widely used for bringing the touch from binary to multiple states, creating new abilities on surface interactions. However, prior proposed force sensing techniques mainly focus on enabling force-applied gestures on certain devices. This paper presents Nail+, a technique using fingernail deformation to enable force touch sensing interactions on everyday rigid surfaces. Our prototype, 3x3 0.2mm strain sensor array mounted on a fingernail, was implemented and conducted with a 12-participant study for evaluating the feasibility of this sensing approach. Result showed that the accuracy for sensing normal and force-applied tapping and swiping can achieve 84.67% on average. We finally proposed two example applications using Nail+ prototype for controlling the interfaces of head-mounted display (HMD) devices and remote screens.

The rise of smartwatches calls for efficient, convenient and suitable text input methods for these small computers. The minuscule size of these screens brings many challenges on how to interact with these devices. Keyboard design requires optimization for these small screens to provide a good user experience and fast text entry method on these devices. We introduce SwipeKey, a text input method that uses swipe directions to allow multiple inputs per button and thus allows for an increase in the effective size of input buttons. We have conducted thorough experiments optimizing SwipeKey to create a fast, low-error, and easy to learn soft keyboard for smartwatches. These benefits result from having a keyboard that emphasizes the use of swipe motions. Our user study results show that with a specific combination of swipe directions and corresponding button size, SwipeKey users achieved a speed of 11 in words per minute (WPM), a 53% improvement from baseline (7.2 in WPM) and dramatically decreased character error rate (CER) from the baseline of 10% down to 3.4%.

For makers and developers, circuit prototyping is an integral part of building electronic projects. Currently, it is common to build circuits based on breadboard schematics that are available on various maker and DIY websites. Some breadboard schematics are used as is without modification, and some are modified and extended to fit specific needs. In such cases, diagrams and schematics merely serve as blueprints and visual instructions, but users still must physically wire the breadboard connections, which can be time-consuming and error-prone. We present CircuitStack, a system that combines the flexibility of breadboarding with the correctness of printed circuits, for enabling rapid and extensible circuit construction. This hybrid system enables circuit reconfigurability, component reusability, and high efficiency at the early stage of prototyping development.

This work presents WonderLens, a system of optical lenses and mirrors for enabling tangible interactions on printed paper. When users perform spatial operations on the optical components, they deform the visual content that is printed on paper, and thereby provide dynamic visual feedback on user interactions without any display devices. The magnetic unit that is embedded in each lens and mirror allows the unit to be identified and tracked using an analog Hall-sensor grid that is placed behind the paper, so the system provides additional auditory and visual feedback through different levels of embodiment, further enhancing the interactivity with the printed content on the physical paper.

We present FlickBoard, which combines a touchpad and a keyboard into the same interaction area to reduce hand movement between a separate keyboard and touchpad. Our main contribution is automatic mode switching between typing and pointing, and the first system capable of combining a trackpad and a keyboard into an single interaction area without the need for external switches. We developed a prototype by embedding a 58x20 capacitive sensing grid into a soft keyboard cover, and used machine learning to distinguish between moving a cursor (touchpad mode) and entering text (keyboard mode). We conducted experimental studies that show automatic mode switching classification accuracies of 98% are achievable with our technology. Finally, our prototype has a thin profile and can be placed over existing keyboards.

Smart glasses, such as Google Glass, provide always-available displays not offered by console and mobile gaming devices, and could potentially offer a pervasive gaming experience. However, research on input for games on smart glasses has been constrained by the available sensors to date. To help inform design directions, this paper explores user-defined game input for smart glasses beyond the capabilities of current sensors, and focuses on the interaction in public settings. We conducted a user-defined input study with 24 participants, each performing 17 common game control tasks using 3 classes of interaction and 2 form factors of smart glasses, for a total of 2448 trials. Results show that users significantly preferred non-touch and non-handheld interaction over using handheld input devices, such as in-air gestures. Also, for touch input without handheld devices, users preferred interacting with their palms over wearable devices (51% vs 20%). In addition, users preferred interactions that are less noticeable due to concerns with social acceptance, and preferred in-air gestures in front of the torso rather than in front of the face (63% vs 37%).

With the advent of the technological era, Computer Vision Syndrome (CVS) has become a common problem. Due to the small hand-held form of smartphones, many users are not aware of the distance between eyes and the device. This habit may lead to negative impacts on ocular health. Always keeping a safe distance prevents the user from CVS. In this paper, we present a mobile application that utilizes the front camera to estimate the distance between the phone and the user. Once this distance is too short, the system alerts the user. We explore four different types of warnings: Blur, Flashing Border, Spot, and Push Notification. We conducted user studies to measure satisfaction, effectiveness, and distraction of these warning methods. Our results show that our system is effective in reminding users to keep a safe distance.

We present PalmType, which uses palms as interactive keyboards for smart wearable displays, such as Google Glass. PalmType leverages users' innate ability to pinpoint specific areas of their palms and fingers without visual attention (i.e. proprioception), and provides visual feedback via the wearable displays. With wrist-worn sensors and wearable displays, PalmType enables typing without requiring users to hold any devices and does not require visual attention to their hands. We conducted design sessions with 6 participants to see how users map QWERTY layout to their hands based on their proprioception. To evaluate typing performance and preference, we conducted a 12-person user study using Google Glass and Vicon motion tracking system, which showed that PalmType with optimized QWERTY layout is 39% faster than current touchpad-based keyboards. In addition, PalmType is preferred by 92% of the participants. We demonstrate the feasibility of wearable PalmType by building a prototype that uses a wrist-worn array of 15 infrared sensors to detect users' finger position and taps, and provides visual feedback via Google Glass.

In this paper, we explored eyes-free gesture interactions on palms, which enables users to interact with devices by drawing stroke gestures on palms without looking at palms. We conducted a 24-person user study to understand how users draw gestures on the palm with varying characteristics including regions, orientation and starting points. Based on the findings, we proposed two new interaction techniques for palm-based gesture interface. To explore and demonstrate the feasibility of the interaction, we implemented EyeWrist, a wrist-mounted prototype which detects gestures on palms by using an IR camera and laser-line projector. The preliminary evaluation revealed that EyeWrist enabled users to draw graffiti letter and multi-stroke gestures with above 90% accuracy and that both the concept of using palms as gesture interfaces for eyes-free input and the proposed two interaction techniques were appealing to users.

For live music performances, current Wireless In-Ear Monitoring and Personal Mixing setups require a lot of equipment and wiring. This paper introduces Giggler, a system that makes a Wireless In-Ear Monitoring and Personal Mixing experience, easier to setup, easier to use, faster to control and more accessible for musicians than conventional systems by integrating all equipment into one mobile device per musician. The results of the two user studies that we conducted show that Giggler's User Interface performs up to more than twice as fast as a traditional mixer and indicate that Giggler outperforms a Physical mixer because visual features are added to streamline the channel identification process.

In this paper, we explore using the back of hands for sensing hand gestures, which interferes less than glove-based approaches and provides better recognition than sensing at wrists and forearms. Our prototype, BackHand, uses an array of strain gauge sensors affixed to the back of hands, and applies machine learning techniques to recognize a variety of hand gestures. We conducted a user study with 10 participants to better understand gesture recognition accuracy and the effects of sensing locations. Results showed that sensor reading patterns differ significantly across users, but are consistent for the same user. The leave-one-user-out accuracy is low at an average of 27.4%, but reaches 95.8% average accuracy for 16 popular hand gestures when personalized for each participant. The most promising location spans the 1/8~1/4 area between the metacarpophalangeal joints (MCP, the knuckles between the hand and fingers) and the head of ulna (tip of the wrist).

We present TouchSense, which provides additional touchscreen input vocabulary by distinguishing the areas of users' finger pads contacting the touchscreen. It requires minimal touch input area and minimal movement, making it especially ideal for wearable devices such as smart watches and smart glasses. For example, users of a calculator application on a smart watch could tap normally to enter numbers, and tap with the right side of their fingers to enter the operators (e.g. , -, =). Results from two human-factor studies showed that users could tap a touchscreen with five or more distinct areas of their finger pads. Also, they were able to tap with more distinct areas closer to their fingertips. We developed a TouchSense smart watch prototype using inertial measurement sensors, and developed two example applications: a calculator and a text editor. We also collected user feedback via an explorative study.

We present EverTutor, a system that automatically generates interactive tutorials on smartphone from user demonstration. For tutorial authors, it simplifies the tutorial creation. For tutorial users, it provides contextual step-by-step guidance and avoids the frequent context switching between tutorials and users' primary tasks. In order to generate the tutorials automatically, EverTutor records low-level touch events to detect gestures and identify on-screen targets. When a tutorial is browsed, the system uses vision-based techniques to locate the target regions and overlays the corresponding input prompt contextually. It also identifies the correctness of users' interaction to guide the users step by step. We conducted a 6-person user study for creating tutorials and a 12-person user study for browsing tutorials, and we compared EverTutor's interactive tutorials to static and video ones. Study results show that creating tutorials by EverTutor is simpler and faster than producing static and video tutorials. Also, when using the tutorials, the task completion time for interactive tutorials were 3-6 times faster than static and video tutorials regardless of age group. In terms of user preference, 83% of the users chose interactive type as the preferred tutorial type and rated it easiest to follow and easiest to understand.

Smart Glasses offer the opportunity to use head mounted sensors, such as gyroscope and accelerometers, to enable new types of game interaction. To better understand game play experience on Smart Glasses, we recruited 24 participants to play four current games on Google Glass that uses different interaction methods, including gyroscope, voice, touchpad, and in-air gesture. Study results showed that participants were concerned with comfort and social acceptance. Also, their favorite input method was gyroscope, and their favorite game type was First-Person Shooter (FPS) game. Hence, we implemented a FPS game on Google Glass using gyroscope for changing the viewport, and divide FPS controls into four categories: (a)Viewport Control, (b)Aim Control, (c)Fire Control, (d) Move Control. We implemented multiple control method in each category to evaluate and explore glass game control design.

Multitouch tablets, such as iPad and Android tablets, support virtual keyboards for text entry. Our 64-user study shows that 98% of the users preferred different keyboard layouts and positions depending on how they were holding these devices. However, current tablets either do not allow keyboard adjustment or require users to manually adjust the keyboards. We present iGrasp, which automatically adapts the layout and position of virtual keyboards based on how and where users are grasping the devices without requiring explicit user input. Our prototype uses 46 capacitive sensors positioned along the sides of an iPad to sense users' grasps, and supports two types of grasp-based automatic adaptation: layout switching and continuous positioning. Our two 18-user studies show that participants were able to begin typing 42% earlier using iGrasp's adaptive keyboard compared to the manually adjustable keyboard. Participants also rated iGrasp much easier to use than the manually adjustable keyboard (4.2 vs 2.9 on five-point Likert scale.)

Automatic screen rotation improves viewing experience and usability of mobile devices, but current gravity-based approaches do not support postures such as lying on one side, and manual rotation switches require explicit user input. iRotateGrasp automatically rotates screens of mobile devices to match users' viewing orientations based on how users are grasping the devices. Our insight is that users' grasps are consistent for each orientation, but significantly differ between different orientations. Our prototype used a total of 44 capacitive sensors along the four sides and the back of an iPod Touch, and uses support vector machine (SVM) to recognize grasps at 25Hz. We collected 6-users' usage under 108 different combinations of posture, orienta-tion, touchscreen operation, and left/right/both hands. Our offline analysis showed that our grasp-based approach is promising, with 80.9% accuracy when training and testing on different users, and up to 96.7% if users are willing to train the system. Our user study (N=16) showed that iRo-tateGrasp had an accuracy of 78.8% and was 31.3% more accurate than gravity-based rotation.

We present GaussBits, which is a system of the passive magnetic tangible designs that enables 3D tangible interactions in the near-surface space of portable displays. When a thin magnetic sensor grid is attached to the back of the display, the 3D position and partial 3D orientation of the GaussBits can be resolved by the proposed bi-polar magnetic field tracking technique. This portable platform can therefore enrich tangible interactions by extending the design space to the near-surface space. Since non-ferrous materials, such as the user's hand, do not occlude the magnetic field, interaction designers can freely incorporate a magnetic unit into an appropriately shaped non-ferrous object to exploit the metaphors of the real-world tasks, and users can freely manipulate the GaussBits by hands or using other non-ferrous tools without causing interference. The presented example applications and the collected feedback from an explorative workshop revealed that this new approach is widely applicable.

We present RealSense, a technology that enables users to easily share media files with proximate users by performing directional gestures on mobile devices. RealSense leverages the natural human group behavior of forming a circle and facing the center of the group. By continuously monitoring the directional heading of each device using only built-in orientation sensors, RealSense can compute the relative direction between all the devices. It simplifies media sharing because users do not need to lookup and specify the user IDs and device IDs of the intended recipients. We first evaluated the feasibility and design of RealSense, including the orientation sensor error and the minimal arc degree for selecting recipients. We then compared RealSense with three other common sharing interactions: 1) linear menu, 2) pie menu, and 3) NFC. Our results show that participants preferred RealSense over other sharing interactions, especially for groups of participants who were unacquainted with each other.

We present FingerPad, a nail-mounted device that turns the tip of the index finger into a touchpad, allowing private and subtle interaction while on the move. FingerPad enables touch input using magnetic tracking, by adding a Hall sensor grid on the index fingernail, and a magnet on the thumbnail. Since it permits input through the pinch gesture, FingerPad is suitable for private use because the movements of the fingers in a pinch are subtle and are naturally hidden by the hand. Functionally, FingerPad resembles a touchpad, and also allows for eyes-free use. Additionally, since the necessary devices are attached to the nails, FingerPad preserves natural haptic feedback without affecting the native function of the fingertips. Through user study, we analyze the three design factors, namely posture, commitment method and target size, to assess the design of the FingerPad. Though the results show some trade-off among the factors, generally participants achieve 93% accuracy for very small targets (1.2mm-width) in the seated condition, and 92% accuracy for 2.5mm-width targets in the walking condition.

Cascading Style Sheet (CSS) is a fundamental web language for describing the presentation of web pages. CSS rules are often reused across multiple parts of a page and across multiple pages throughout a site to reduce repetition and to provide a consistent look and feel. When a CSS rule is modified, developers currently have to manually track and visually inspect all possible parts of the site that may be impacted by that change. We present SeeSS, a system that automatically tracks CSS change impact across a site and enables developers to easily visualize all of them. The impacted page fragments are sorted by severity and the differences before and after the change are highlighted using animation.

We present iRotate, an approach to automatically rotate screens on mobile devices to match users' face orientation. Current approaches to automatic screen rotation are based on gravity and device orientation. Our survey of 513 users shows that 42% currently experience auto-rotation that leads to incorrect viewing orientation several times a week or more, and 24% find the problem to be very serious to extremely serious. iRotate augments gravity-based approach, and uses front cameras on mobile devices to detect users' faces and rotates screens accordingly. It requires no explicit user input and supports different user postures and device orientations. We have implemented a iRotate that works in real-time on iPhone and iPad, and we assess the accuracy and limitations of iRotate through a 20- participant feasibility study.

Most mobile games are designed for users to only focus on their own screens thus lack of face-to-face interaction even users are sitting together. Prior work shows that the shared information space created by multiple mobile devices can encourage users to communicate to each other naturally. The aim of this work is to provide a fluent view-stitching technique for mobile phone users to establish their information-shared view. We present MagMobile: a new spatial interaction technique that allows users to stitch views by simply putting multiple mobile devices close to each other. We describe the design of spatial-aware sensor module which is low cost and easy to be obtained into phones. We also propose two collaborative games to engage social interactions in the co-located place.

We present TUIC, a technology that enables tangible interaction on capacitive multi-touch devices, such as iPad, iPhone, and 3M's multi-touch displays, without requiring any hardware modifications. TUIC simulates finger touches on capacitive displays using passive materials and active modulation circuits embedded inside tangible objects, and can be used with multi-touch gestures simultaneously. TUIC consists of three approaches to sense and track objects: spatial, frequency, and hybrid (spatial plus frequency). The spatial approach, also known as 2D markers, uses geometric, multi-point touch patterns to encode object IDs. Spatial tags are straightforward to construct and are easily tracked when moved, but require sufficient spacing between the multiple touch points. The frequency approach uses modulation circuits to generate high-frequency touches to encode object IDs in the time domain. It requires fewer touch points and allows smaller tags to be built. The hybrid approach combines both spatial and frequency tags to construct small tags that can be reliably tracked when moved and rotated. We show three applications demonstrating the above approaches on iPads and 3M's multi-touch displays.

We present a system which automatically reduces the ringtone volume as soon as users start picking up their mobile devices to answer incoming phone calls. Our system uses multiple sensors already in mobile phones (e.g. accelerometer, gyroscope, and proximity sensor) to extract 47 features, and uses supported vector machine (SVM) as the classifier to identify the act of picking up a phone. We have collected data under several typical conditions: including walking vs stationary and picking up the phone from inside a bag vs pockets. Our results show that the system can correctly identify users picking up their mobile phones 95% of the time on average.

Virtual keyboards and controls, commonly used on mobile multi-touch devices, occlude content of interest and do not provide tactile feedback. Clip-on Gadgets solve these issues by extending the interaction area of multi-touch devices with physical controllers. Clip-on Gadgets use only conductive materials to map user input on the controllers to touch points on the edges of screens; therefore, they are battery-free, lightweight, and low-cost. In addition, they can be used in combination with multi-touch gestures. We present several hardware designs and a software toolkit, which enable users to simply attach Clip-on Gadgets to an edge of a device and start interacting with it.

In this paper, we explore the challenges in applying and investigate methodologies to improve direct-touch interaction on intangible displays. Direct-touch interaction simplifies object manipulation, because it combines the input and display into a single integrated interface. While traditional tangible display-based direct-touch technology is commonplace, similar direct-touch interaction within an intangible display paradigm presents many challenges. Given the lack of tactile feedback, direct-touch interaction on an intangible display may show poor performance even on the simplest of target acquisition tasks. In order to study this problem, we have created a prototype of an intangible display. In the initial study, we collected user discrepancy data corresponding to the interpretation of 3D location of targets shown on our intangible display. The result showed that participants performed poorly in determining the z-coordinate of the targets and were imprecise in their execution of screen touches within the system. Thirty percent of positioning operations showed errors larger than 30mm from the actual surface. This finding triggered our interest to design a second study, in which we quantified task time in the presence of visual and audio feedback. The pseudo-shadow visual feedback was shown to be helpful both in improving user performance and satisfaction.

This paper presents e-Fovea, a system that combines both multi-resolution camera input and multi-resolution steerable projector output to support large-scale and high-resolution visual monitoring. e-Fovea utilizes a design similar to the human eyes, which provides peripheral vision with a steerable fovea that is in higher resolution. e-Fovea is implemented using a steerable telephoto camera and a wide-angle camera. The telephoto image is displayed using a projector with a steerable mirror, and overlaid on the wide-angle image that is displayed using a second projector.

This paper presents i-m-Space, an interactive multimedia rehabilitation space that helps the post-surgery recovery of breast cancer patients. Our goal is to improve patients' physical therapy and psychological relaxation experience through careful applications of multimedia technology. i-m-Space consists of three types of breathing-based relaxation and three types for interactive exercise-based rehabilitation.

This paper presents a programmable infrared (IR) technique that utilizes invisible, programmable markers to support interaction beyond the surface of a diffused-illumination (DI) multi-touch system. We combine an IR projector and a standard color projector to simultaneously project visible content and invisible markers. Mobile devices outfitted with IR cameras can compute their 3D positions based on the markers perceived. Markers are selectively turned off to support multi-touch and direct on-surface tangible input. The proposed techniques enable a collaborative multi-display multi-touch tabletop system. We also present three interactive tools: i-m-View, i-m-Lamp, and i-m-Flashlight, which consist of a mobile tablet and projectors that users can freely interact with beyond the main display surface. Early user feedback shows that these interactive devices, combined with a large interactive display, allow more intuitive navigation and are reportedly enjoyable to use.

Recent advances in small inexpensive sensors, low-power processing, and activity modeling have enabled applications that use on-body sensing and machine learning to infer people's activities throughout everyday life. To address the growing rate of sedentary lifestyles, we have developed a system, UbiFit Garden, which uses these technologies and a personal, mobile display to encourage physical activity. We conducted a 3-week field trial in which 12 participants used the system and report findings focusing on their experiences with the sensing and activity inference. We discuss key implications for systems that use on-body sensing and activity inference to encourage physical activity.

This paper presents MyExperience, a system for capturing both objective and subjective in situ data on mobile computing activities. MyExperience combines the following two techniques: 1) passive logging of device usage, user context, and environmental sensor readings, and 2) active context-triggered user experience sampling to collect in situ, subjective user feedback. MyExperience currently runs on mobile phones and supports logging of more than 140 event types, including: 1) device usage such as communication, application usage, and media capture, 2) user context such as calendar appointments, and 3) environmental sensing such as Bluetooth and GPS. In addition, user experience sampling can be targeted to moments of interest by triggering off sensor readings. We present several case studies of field deployments on people's personal phones to demonstrate how MyExperience can be used effectively to understand how people use and experience mobile technology.

To evaluate ubiquitous computing technologies, which may be embedded in the environment, embedded in objects, worn, or carried by the user throughout everyday life, it is essential to use methods that accommodate the often unpredictable, real-world environments in which the technologies are used. This article discusses how we have adapted and applied traditional methods from psychology and human-computer interaction, such as Wizard of Oz and Experience Sampling, to be more amenable to the in situ evaluations of ubiquitous computing applications, particularly in the early stages of design. The way that ubiquitous computing technologies can facilitate the in situ collection of self-report data is also discussed. Although the focus is on ubiquitous computing applications and tools for their assessment, it is believed that the in situ evaluation tools that are proposed will be generally useful for field trials of other technology, applications, or formative studies that are concerned with collecting data in situ.

Location-based computing (LBC) is becoming increasing important in both industry and academia. A key challenge is the pervasive deployment of LBC technologies; to be effective they must run on a wide variety of client platforms, including laptops, PDAs, and mobile phones, so that location data can be acquired anywhere and accessed by any application. Moreover, as a nascent area, LBC is experiencing rapid innovation in sensing technologies, the positioning algorithms themselves, and the applications they support. Lastly, as a newcomer, LBC must integrate with existing communications and application technologies, including web browsers and location data interchange standard.This paper describes our experience in developing the Place Lab architecture, a widely used first-generation open source toolkit for client-side location sensing. Using a layered, pattern-based architecture, it supports modular development in any dimension of LBC, enabling the field to move forward more rapidly as these innovations are shared with the community as pluggable components. Our experience shows the benefits of domain-specific abstractions, and how we overcame high-level language constraints to support a wide array of platforms in this emerging space. We also describe our experience in re-engineering parts of the architecture based on the needs of the user community, including insights on software licensing issues.

This paper presents Virgil, an automatic access point discovery and selection system. Unlike existing systems that select access points based entirely on received signal strength, Virgil scans for all available APs at a location, quickly associates to each, and runs a battery of tests to estimate the quality of each AP's connection to the Internet. Virgil also probes for blocked or redirected ports, to guide AP selection in favor of preserving application services that are currently in use. Results of our evaluation across five neighborhoods in three cities show Virgil finds a usable connection from 22% to 100% more often than selecting based on signal strength alone. By caching AP test results, Virgil both improves performance and success rate. Our overhead is acceptable and is shown to be faster than manually selecting an AP with Windows XP.

Navigation services (e.g., in-car navigation systems and online mapping sites) compute routes between two locations to help users navigate. However, these routes may direct users along an unfamiliar path when a familiar path exists, or, conversely, may include redundant information that the user already knows. These overly complicated directions increase the cognitive load of the user, which may lead to a dangerous driving environment. Since the level of detail is user specific and depends on their familiarity with a region, routes need to be personalized. We have developed a system, called MyRoute, that reduces route complexity by creating user specific routes based on a priori knowledge of familiar routes and landmarks. MyRoute works by compressing well known steps into a single contextualized step and rerouting users along familiar routes.

In this article, we describe the design process of Reno, a location-enhanced, mobile coordination tool and person finder. The design process included three field experiments: a formative Experience Sampling Method (ESM) study, a pilot deployment and an extended user study. These studies were targeted at the significant personal security, privacy and data protection concerns caused by this application. We distil this experience into a small set of guidelines for designers of social mobile applications and show how these guidelines can be applied to a different application, called Boise. These guidelines cover issues pertaining to personal boundary definition, control, deception and denial, and group vs. individual communication. We also report on lessons learned from our evaluation experience, which might help practitioners in designing novel mobile applications, including the choice and characterization of users for testing security and privacy features of designs, the length of learning curves and their effect on evaluation and the impact of peculiar deployment circumstances on the results of these finely tuned user studies.

Traditional problem determination techniques rely on static dependency models that are difficult to generate accurately in today's large, distributed, and dynamic application environments such as e-commerce systems. We present a dynamic analysis methodology that automates problem determination in these environments by 1) coarse-grained tagging of numerous real client requests as they travel through the system and 2) using data mining techniques to correlate the believed failures and successes of these requests to determine which components are most likely to be at fault. To validate our methodology, we have implemented Pinpoint, a framework for root cause analysis on the J2EE platform that requires no knowledge of the application components. Pinpoint consists of three parts: a communications layer that traces client requests, a failure detector that uses traffic-sniffing and middleware instrumentation, and a data analysis engine. We evaluate Pinpoint by injecting faults into various application components and show that Pinpoint identifies the faulty components with high accuracy and produces few false-positives.

We propose a new connection management architecture for clustered Internet services called active connection management (ACM) to improve the availability, quality of service, and manageability of Internet services. ACM extends the API of load-balancing switches to include application-level primitives. These primitives enable Internet services to manage load-balancing switches dynamically to control how client connections are mapped to physical resources as they enter the cluster. ACM provides several properties ideal for Internet services over static configuration. First, ACM increases availability through automated configuration, rolling reboots, and reduced server failure detection and recovery time. Second, ACM provides better load conditioning by enabling dynamic allocation of front-end resources and enabling more control over graceful degradation. Third, it provides better manageability by automating configuration, reducing human errors and improving scalability. We present a design, implementation, and evaluation of an ACM system, called COMICS (COnnection Manager for Internet-Centric Services), that enables services to actively control how connections are mapped to servers. We believe that ACM primitives should be implemented into future switches and be utilized by application servers to build more robust Internet service platforms.