Instrumental music conveys rich affective experiences through acoustic cues, yet instrumental passages often remain inaccessible to Deaf and Hard-of-Hearing (DHH) audiences. Although captioning practices for vocal songs have expanded, instrumental music remains largely uncaptioned, with no established criteria for representing musical content in text. We propose µCap (Music Captions), an automatic instrumental music captioning system that transforms instrumental audio into time-aligned, non-lexical textual renderings enhanced with simple visuals. Drawing on Preliminary surveys with DHH individuals and expert group discussions, we developed a phonetic-like captioning schema grounded in music sound analysis and linguistics. We then implemented µCap using audio feature extraction and a retrieval-augmented generation pipeline to produce expressive, sound-mimetic captions. Two user evaluations with DHH participants (n=20 and n=15) showed that µCap enhanced music appreciation, immersion, and perceived presence of acoustic detail. This work contributes empirical evidence and insights for designing caption-based visual representations that make instrumental music more accessible.

Clinical pre-consultation, where patients share health information prior to an appointment, can offer better patient-centered care by freeing time for meaningful patient–physician conversations. Conversational agents powered by large language models (LLMs) can automate this process to make it more scalable and consistent, but doing so can cause information overload that exacerbates physicians’ workload as they spend time parsing through data. This paper examines the opportunities and challenges of using pre-consultation agents to mediate information transfer between patients and physicians, with the aim of producing clinically useful yet patient-driven summaries that capture their histories and concerns. Through sessions with both patients and physicians, we show that automatically generated pre-consultation summaries can increase patients’ confidence and sense of control over their health information while fostering a more collaborative dynamic. We conclude with design recommendations for integrating pre-consultation agents into clinical workflows.

d/Deaf and Hard of Hearing (DHH) individuals often engage with music through a multimodal approach, where visual modalities are also used rather than relying on sound alone. While tools like captions and visualizers offer partial support, they often fail to capture the emotional depth and structural nuances of music. To explore new possibilities, we adopted an iterative, probe-based approach. Through a formative study with 9 DHH participants, we identified key design requirements for visualizing rhythm, emotion, and lyrics. We developed FAME (Facial Avatar for Musical Expression), a design probe that conveys music through expressive facial animation, instrument highlights, and synchronized captions, lip-syncing to lyrics or scat-singing to melodies. Through a two-phase exploratory study with 12 DHH users, we examined FAME’s efficacy, applicability, and requirements for representing musical elements. Our findings refine design requirements for avatar-based systems and highlight the potential of avatars as expressive and socially meaningful tools for music accessibility.

Affective and prosodic captions convey not only what a speaker says, but also how they say it-louder words may appear thicker, quieter ones thinner; angry in red, calm in blue.These captions can improve access, satisfaction, and engagement for d/Deaf and Hard-of-Hearing (dhh) users.While prior work has explored their design space, it has focused largely on dhh participants in North America, limiting generalizability beyond English and Latin-based scripts.To uncover the role of culture and language, we ran an exploratory study with 49 dhh participants from North America and South Korea using CuCap, a tool that allowed them to personalize which speech features were displayed, and how.While emotion visualization was a universally favored choice, confirming prior findings, prosody preferences varied across cultures, reflecting linguistic and hearing factors.These findings point to the need for flexible captioning systems that account for cultural, linguistic, and individual differences.

Pre-consultation serves as a critical information exchange between healthcare providers and patients, streamlining visits and supporting patient-centered care. Human-led pre-consultations offer many benefits, yet they require significant time and energy from clinical staff. In this work, we identify design goals for pre-consultation chatbots given their potential to carry out human-like conversations and autonomously adapt their line of questioning. We conducted a study with 33 walk-in clinic patients to elicit design considerations for pre-consultation chatbots. Participants were exposed to one of two study conditions: an LLM-powered AI agent and a Wizard-of-Oz agent simulated by medical professionals. Our study found that both conditions were equally well-received and demonstrated comparable conversational capabilities. However, the extent of the follow-up questions and the amount of empathy impacted the chatbot’s perceived thoroughness and sincerity. Patients also highlighted the importance of setting expectations for the chatbot before and after the pre-consultation experience.

Menstrual trackers currently lack the affordances required to help individuals achieve their goals beyond menstrual event predictions and symptom logging. Taking an initial step towards this aspiration, we propose, validate, and refine five functional design requirements for future interface designs that facilitate menstrual data exploration. We interviewed 30 individuals who menstruate and collected their feedback on the practical application of these requirements. To elicit ideas and impressions, we designed two proof-of-concept interfaces to use as design probes with similar core functionalities but different presentations of phase timing predictions and signal arrangement. Our analysis revealed participants’ feedback regarding the presentation of predictions for menstrual-related events, the visualization of future signal patterns, personalization abilities for viewing signals relevant to their menstrual experience, the availability of resources to understand the underlying biological connections between signals, and the ability to compare multiple cycles side-by-side with context.

Creating dedicated accounts to post users’ pet content is a growing trend on Instagram. While these account owners derive joy from this pursuit, they may also struggle with criticisms and challenges. Yet, there remains a knowledge gap on how pet account owners manage their pets’ online presence and navigate these obstacles successfully. Drawing from interviews with 21 Instagram pet account owners, we uncover the motivations behind pet account creation, spanning personal, altruistic, and commercial goals. We learn about the strategies employed for crafting their pets’ online identities and personas, as well as the challenges faced by both owners and their pets in navigating the complexities of digital identity management. We discuss the evolving dynamics between humans and their pets, positioning pet identity cultivation as a form of collaborative work, akin to the “third shift”, highlighting the need to design interfaces that support this unique identity management process.

Medical consultations over synchronous text-based platforms are becoming increasingly popular for virtual care, yet little is known about how physicians translate their training to this healthcare medium. We report the constraints, workarounds, and opportunities highlighted by eight primary care physicians who used such a platform in simulated medical scenarios with standardized patients. We found that due to the perceived inefficiency of communicating over text, the physicians made subconscious use of double-barreled questions and action multiplexing to streamline the conversation. In addition, the physicians overcame the lack of missing verbal and visual cues by adding explicit messages to convey empathy and active listening. We also identify several affordances of text-based platforms, such as the ability for users to reference the conversation history and for patients to feel a sense of privacy during sensitive disclosure. From these findings, we propose design opportunities for how future synchronous text-based platforms can better support medical consultations.

Following along how-to videos requires alternating focus between understanding procedural video instructions and performing them. Examining how to support these continuous context switches for the user has been largely unexplored. In this paper, we describe a user study with thirty participants who performed an hour-long cooking task while interacting with a wizard-of-oz hands-free interactive system that is aware of both their cooking progress and environment contexts. Through analysis of the session scripts, we identify a dichotomy between participant query differences and workflow alignment similarities, under-studied interactions that require AI functionality beyond video navigation alone, and queries that call for multimodal sensing of a user’s environment. By understanding the assistant experience through the participants’ interactions, we identify design implications for a smart assistant that can discern a user’s task completion flow and personal characteristics, accommodate requests within and external to the task domain, and support nonvoice-based queries.

Song signing is a method practiced by people who are d/Deaf and non-d/Deaf individuals to visually represent music and make music accessible through sign language and body movements. Although there is growing interest in song signing, there is a lack of understanding on what d/Deaf people value about song signing and how to make song signing productions that they would consider acceptable. We conducted semi-structured interviews with 12 d/Deaf participants to gain a deeper understanding of what they value in music and song signing. We then interviewed 14 song signers to understand their experiences and processes in creating song signing performances. From this study, we identify three complex, interrelated layers of the song signing creation process and discuss how they can be supported and completed to potentially bridge the cultural divide between the d/Deaf and non-d/Deaf audiences and guide more culturally responsive creation of music.

AAC research has traditionally focused on input speed, leaving higher-level communication goals such as relational maintenance under-explored. Through semi-structured interviews with AAC users with motor and speech impairments and their primary family caregivers, we offer a nuanced understanding of AAC’s roles in maintaining close relationships. Our inductive analysis reveals emerging themes including how AAC users and their partners share the physical and mental workload to overcome communication barriers in complex situations. Our deductive application of the Relational Maintenance Strategies framework exposes the efforts made and the challenges encountered in managing social engagements, providing mutual support, and decoding implicit expressions. From these insights, we propose novel research directions for better supporting maintenance strategies and social purposes of communication, including notably mediating relational tensions, leveraging empowerment and identity, and supporting interactions for social closeness and etiquette, which we hope will motivate discussion in HCI communities on expanding AAC research space.

Deaf and Hard-of-Hearing (DHH) audiences have long complained about caption qualities for many online videos created by individual content creators on video-sharing platforms (e.g., YouTube). However, there lack explorations of practices, challenges, and perceptions of online video captions from the perspectives of both individual content creators and DHH audiences. In this work, we first explore DHH audiences' feedback on and reactions to YouTube video captions through interviews with 13 DHH individuals, and uncover DHH audiences' experiences, challenges, and perceptions on watching videos created by individual content creators (e.g., manually added caption tags could create additional confidence and trust in caption qualities for DHH audiences). We then discover individual content creators' practices, challenges, and perceptions on captioning their videos (e.g., back-captioning problems) by conducting a YouTube video analysis with 189 captioning-related YouTube videos, followed by a survey with 62 individual content creators. Overall, our findings provide an in-depth understanding of captions generated by individual content creators and bridge the knowledge gap mutually between content creators and DHH audiences on captions.

People with chronic obstructive pulmonary disease (COPD) experience dyspnea and dyspnea-related distress and anxiety (DDA) upon physical exertion. When performing supervised exercise, measurement of physiological data, such as heart rate (HR) and blood oxygen saturation (O2sat), are commonly used for safety, but the impacts of such monitoring on their perceptions and behaviour have not previously been studied. This paper investigates the effect of presenting live physiological data to people with COPD during exercise with a focus on its impact on perceptions of dyspnea intensity (DI) and DDA. Informed by formative interviews with 15 people with COPD, we design VIDDE, an exercise companion tool visualizing live data from a pulse oximeter, and evaluate its effect on DI and DDA through case studies involving 3 participants with COPD exercising at their homes. We also conducted design probe interviews with 6 more participants with COPD to investigate their needs and design requirements for an exercise-companion application that featured physiological data monitoring. Our results suggest that presenting live physiological data during exercise is of value, and can contribute to reduced DDA, better understanding of breathlessness sensations, while providing sufficient reassurance to encourage physical activity.

Despite the potential benefits of assistive technologies (ATs) for people with various disabilities, only around 7% of Chinese with disabilities have had an opportunity to use ATs. Even for those who have used ATs, the abandonment rate was high. Although China has the world's largest population with disabilities, prior research exploring how ATs are used and perceived, and why ATs are abandoned have been conducted primarily in North America and Europe. In this paper, we present an interview study conducted in China with 26 people with various disabilities to understand their practices, challenges, perceptions, and misperceptions of using ATs. From the study, we learned about factors that influence AT adoption practices (e.g., misuse of accessible infrastructure, issues with replicating existing commercial ATs), challenges using ATs in social interactions (e.g., Chinese stigma), and misperceptions about ATs (e.g., ATs should overcome inaccessible social infrastructures). Informed by the findings, we derive a set of design considerations to bridge the existing gaps in AT design (e.g., manual vs. electronic ATs) and to improve ATs' social acceptability in China.

Recently, digital scribe systems have been gaining popularity as a possible work-around solution to the Electronic Medical Record (EMR) documentation burden that affects many physicians. The proposed system would automate the clinical summary physicians take by capturing and extracting the patient-physician conversation during the consultation. While promising in concept, how this system would apply to real-world use and its limitations are still not well understood. To examine these issues, we designed a digital scribe prototype to generate notes of different qualities ranging from the reality of current state-of-the-art technology to the potential of future implementations. We conducted a ”Wizard of Oz” study with 24 primary care physicians using our digital scribe prototype in 4 simulated medical encounters followed by a semi-structured interview. This exploratory study provides an understanding of physicians’ interaction with digitally scribed notes, their perceptions on note quality, their perceived workflow impact and several directions for improvements.

Chronic pain is often an ongoing challenge for patients to track and collect data. Pain-O-Vision is a smartwatch enabled pain management system that uses computer vision to capture the details of painful events from the user. A natural reaction to pain is to clench ones fist. The embedded camera is used to capture different types of fist clenching, to represent different levels of pain. An initial prototype was built on an Android smartwatch that uses a cloud-based classification service to detect the fist clench gestures. Our results show that it is possible to map a fist clench to different levels of pain which allows the patient to record the intensity of a painful event without carrying a specialized pain management device.

Think-aloud protocols are a highly valued usability testing method for identifying usability problems. Despite the value of conducting think-aloud usability test sessions, analyzing think-aloud sessions is often time-consuming and labor-intensive. Consequently, previous research has urged the community to develop techniques to support fast-paced analysis. In this work, we took the first step to design and evaluate machine learning (ML) models to automatically detect usability problem encounters based on users’ verbalization and speech features in think-aloud sessions. Inspired by recent research that shows subtle patterns in users’ verbalizations and speech features tend to occur when they encounter problems, we examined whether these patterns can be utilized to improve the automatic detection of usability problems. We first conducted and recorded think-aloud sessions and then examined the effect of different input features, ML models, test products, and users on usability problem encounters detection. Our work uncovers several technical and user interface design challenges and sets a baseline for automating usability problem detection and integrating such automation into UX practitioners’ workflow.

Persons with dementia and their care partners have been found to adapt their own technological arrangements using commercially available information and communication technologies (ICTs). Yet, little is known about these processes of technology appropriation and how care practices are impacted. Adopting a relational perspective of care, we longitudinally examined how four family care networks appropriated a new commercial ICT service into their existing technological arrangements and care practices. Cross-case analysis interpreted collaborative appropriation to encompass two interrelated processes of creating and adapting technological practices and negotiating and augmenting care relationships . Four driving forces were also proposed: motivating meanings that actors ascribe to the technology and its use; the learnability of the technology and actors’ resourcefulness ; the establishment of responsive and cooperative care practices ; and the qualities of empathy and shared power in care relationships . The importance of technological literacy, learning, meaning-making, and the nature and quality of care relationships are discussed. Future work is urged to employ longitudinal and naturalistic approaches, and focus design efforts on promoting synergistic care relationships and care practices.

Think-aloud protocols are widely used by user experience (UX) practitioners in usability testing to uncover issues in user interface design. It is often arduous to analyze large amounts of recorded think-aloud sessions and few UX practitioners have an opportunity to get a second perspective during their analysis due to time and resource constraints. Inspired by the recent research that shows subtle verbalization and speech patterns tend to occur when users encounter usability problems, we take the first step to design and evaluate an intelligent visual analytics tool that leverages such patterns to identify usability problem encounters and present them to UX practitioners to assist their analysis. We first conducted and recorded think-aloud sessions, and then extracted textual and acoustic features from the recordings and trained machine learning (ML) models to detect problem encounters. Next, we iteratively designed and developed a visual analytics tool, VisTA, which enables dynamic investigation of think-aloud sessions with a timeline visualization of ML predictions and input features. We conducted a between-subjects laboratory study to compare three conditions, i.e., VisTA, VisTASimple (no visualization of the ML's input features), and Baseline (no ML information at all), with 30 UX professionals. The findings show that UX professionals identified more problem encounters when using VisTA than Baseline by leveraging the problem visualization as an overview, anticipations, and anchors as well as the feature visualization as a means to understand what ML considers and omits. Our findings also provide insights into how they treated ML, dealt with (dis)agreement with ML, and reviewed the videos (i.e., play, pause, and rewind).

The concurrent think-aloud protocol—in which participants verbalize their thoughts when performing tasks—is a widely employed approach in usability testing. Despite its value, analyzing think-aloud sessions can be onerous because it often entails assessing all of a user's verbalizations. This has motivated previous research on developing categories to segment verbalizations into manageable units of analysis. However, the way in which a category might relate to usability problems is currently unclear. In this research, we sought to address this gap in our understanding. We also studied how speech features might relate to usability problems. Through two studies, this research demonstrates that certain patterns of verbalizations are more telling of usability problems than others and that these patterns are robust to different types of test products (i.e., physical devices and digital systems), access to different types of information (i.e., video and audio modality), and the presence or absence of a visualization of verbalizations. The implication is that the verbalization and speech patterns can potentially reduce the time and effort required for analysis by enabling evaluators to focus more on the important aspects of a user's verbalizations. The patterns could also potentially be used to inform the design of systems to automatically detect when in the recorded think-aloud sessions users experience problems.

In this paper, we present BrailleSketch, a gesture-based text input method on touchscreen smartphones for people with visual impairments. To input a letter with BrailleSketch, a user simply sketches a gesture that passes through all dots in the corresponding Braille code for that letter. BrailleSketch allows users to place their fingers anywhere on the screen to begin a gesture and draw the Braille code in many ways. To encourage users to type faster, BrailleSketch does not provide immediate letter-level audio feedback but instead provides word-level audio feedback. It uses an auto-correction algorithm to correct typing errors. Our evaluation of the method with ten participants with visual impairments who each completed five typing sessions shows that BrailleSketch supports a text entry speed of 14.53 word per min (wpm) with 10.6% error. Moreover, our data suggest that the speed had not begun to plateau yet by the last typing session and can continue to improve. Our evaluation also demonstrates the positive effect of the reduced audio feedback and the auto-correction algorithm.

Haptic learning of gesture shortcuts has never been explored. In this paper, we investigate haptic learning of a freehand semaphoric finger tap gesture shortcut set using haptic rings. We conduct a two-day study of 30 participants where we couple haptic stimuli with visual and audio stimuli, and compare their learning performance with wholly visual learning. The results indicate that with <30 minutes of learning, haptic learning of finger tap semaphoric gestures is comparable to visual learning and maintains its recall on the second day.

In this paper, we present SoQr, a sensor that can be attached to an external surface of a household item to estimate the amount of content inside it. The sensor consists of a speaker and a microphone. It outputs a short duration sine wave probing sound to excite a container and its content, and then records the container's impulse response. SoQr then extracts Mean Mel-Frequency Cepstral Coefficients from impulse response recordings of a container with different content levels and learns a support vector machine classifier. Results from a 10-fold cross validation of the prediction models on 19 common household items demonstrate that SoQr can correctly estimate the content level for these products with an average overall F-Measure above 0.96. We then further evaluated SoQr's robustness in different usage scenarios to gain an understanding of how the system performs and specific challenges that might arise when users interact with these products and the sensor.

Unlock gestures are performed by billions of users across the world multiple times a day. Beyond preventing accidental input on mobile devices, they currently serve little to no other purpose. In this paper, we explore how replacing the regular unlock screen with one that asks the user to perform a simple, optional task, can benefit a wealth of application domains, including data collection, personal-health metrics collection, and human intelligence tasks. We evaluate this concept, which we refer to as Slide to X. Further, we show that people are willing to perform microtasks presented through this interface and continue to do so throughout the day while they visit different locations as part of their daily routines. We then discuss how to implement this concept and demonstrate three applications.

We introduce a distinction between disambiguation supporting continuous versus discrete ambiguous text entry. With continuous ambiguous text entry methods, letter selections are treated as ambiguous due to expected imprecision rather than due to discretized letter groupings. We investigate the simple case of a one-dimensional character layout to demonstrate the potential of techniques designed for imprecise entry. Our rotation-based sight-free technique, Rotext, maps device orientation to a layout optimized for disambiguation, motor efficiency, and learnability. We also present an audio feedback system for efficient selection of disambiguated word candidates and explore the role that time spent acknowledging word-level feedback plays in text entry performance. Through a user study, we show that despite missing on average by 2.46--2.92 character positions, with the aid of a maximum a posteriori (MAP) disambiguation algorithm, users can average a sight-free entry speed of 12.6wpm with 98.9% accuracy within 13 sessions (4.3 hours). In a second study, expert users are found to reach 21wpm with 99.6% accuracy after session 20 (6.7 hours) and continue to grow in performance, with individual phrases entered at up to 37wpm. A final study revisits the learnability of the optimized layout. Our modeling of ultimate performance indicates maximum overall sight-free entry speeds of 29.0wpm with audio feedback, or 40.7wpm if an expert user could operate without relying on audio feedback.

Sighted individuals often develop significant knowledge about their environment through what they can visually observe. In contrast, individuals who are visually impaired mostly acquire such knowledge about their environment through information that is explicitly related to them. This paper examines the practices that visually impaired individuals use to learn about their environments and the associated challenges. In the first of our two studies, we uncover four types of information needed to master and navigate the environment. We detail how individuals' context impacts their ability to learn this information, and outline requirements for independent spatial learning. In a second study, we explore how individuals learn about places and activities in their environment. Our findings show that users not only learn information to satisfy their immediate needs, but also to enable future opportunities -- something existing technologies do not fully support. From these findings, we discuss future research and design opportunities to assist the visually impaired in independent spatial learning.

Volunteer socializers are members of a community who voluntarily help newcomers become familiar with the popular practices and attitudes of the community. In this paper, we explore the social and psychological predictors of volunteer socializers on reddit, an online social news-sharing community. Through a survey of over 1000 reddit users, we found that social identity, prosocial-orientation and generalized reciprocity are all predictors of socializers in the community. Interestingly, a user's tenure with the online community has a quadratic effect on volunteer socialization behaviors -- new and long-time members are both more likely to help newcomers than those in between. We conclude with design implications for motivating users to help newcomers.

Though rapid depletion of natural resources has become a global problem, most of the solutions developed to address it are based on studies done in the developed world. Moreover, the commercial sector is among the primary consumers of resources, yet research work has been mostly limited to residential users. We present a study exploring employees' perception, their beliefs and attitudes, towards environmental sustainability at workplaces in a developing region. To obtain broader context, we also conducted a focus group with the facility team members. Our study highlights that in spite of strong motivations to conserve, employees conservative actions are limited due to lack of controls, knowledge and responsibility. We identify new opportunities for design such as designing location specific buildings, removing inefficient choices, and building communal spaces, to facilitate conservation at workplaces.

In this article, we describe the origins, use, and efficacy of a contextualized method for evaluating mobile and ubiquitous computing systems. This technique, which we called paratyping, is based on experience prototyping and event-contingent experience sampling and allows researchers to survey people in real-life situations without the need for costly and sometimes untenable deployment evaluations. We used this tool to probe the perceptions of the conversation partners of users of the Personal Audio Loop, a memory aid with the potential for substantial privacy implications. Based on that experience, we refined and adapted the approach to evaluate SenseCam, a wearable, automatic picture-taking device, across multiple geographic locations. We describe the benefits, challenges, and methodological considerations that emerged during our use of the paratyping method across these two studies. We describe how this method blends some of the benefits of survey-based research with more contextualized methods, focusing on trustworthiness of the method in terms of generating scientific knowledge. In particular, this method is a good fit for studying certain classes of mobile and ubiquitous computing applications but can be applied to many types of applications.

Proximate mobile devices that are aware of their orientation relative to one another can support novel and natural forms of interaction. In this paper, we present a method to determine the relative orientation of proximate mobile devices using only the backside camera. We implemented this method as a service called Orienteer, which provides mobile device with the orientation of other proximate mobile devices. We demonstrate that orientation information can be used to enable novel and natural interactions by developing two applications that allow the user to push content in the direction of another device to share it and point the device toward another to filter content based on the device's owner. An informal evaluation revealed that interactions built upon orientation information can be natural and compelling to users, but developers and designers need to carefully consider how orientation should be applied effectively.

An essential aspect of learning a second language is the acquisition of vocabulary. However, acquiring vocabulary is often a protracted process that requires repeated and spaced exposure; which can be difficult to accommodate given the busyness of daily living. In this paper, we explore if a learner can implicitly acquire second language vocabulary through her explicit interactions with her mobile phone (e.g., navigating multiple home screens) using an interface we developed called Vocabulary Wallpaper. In addition, we examine if the type of vocabulary this technique exposes to the learner, whether it is contextually relevant or contextually-independent will influence the learner's rate of vocabulary acquisition. The results of our study show participants were able to use Vocabulary Wallpaper to increase the number of second language vocabulary that they can recognize and recall and their rate of vocabulary acquisition was significantly greater when presented with a contextually relevant vocabulary than a contextually-independent vocabulary.

Visually impaired pedestrians experience unique challenges when navigating an urban environment because many cues about orientation and traffic patterns are difficult to ascertain without the use of vision. Technological aids such as customized GPS navigation tools offer the chance to augment visually impaired pedestrians' sensory information with a richer depiction of an environment, but care must be taken to balance the need for more information with other demands on the senses. In this paper, we focus on the information needs of visually impaired pedestrians at intersections, which present a specific cause of stress when navigating in unfamiliar locations. We present a navigation application prototype called CrossingGuard that provides rich information to a user such as details about intersection geometry that are not available to visually impaired pedestrians today. A user study comparing content-rich information to a baseline condition shows that content-rich information raises the level of comfort that visually impaired pedestrians feel at unfamiliar intersections. In addition, we discuss the categories of information that are most useful. Finally, we introduce a micro-task approach to gather intersection data via Street View annotations that achieves 85.5% accuracy over the 9 categories of information used by CrossingGuard.

Learning an environment can be challenging for people with visual impairments. Braille maps allow their users to understand the spatial relationship between a set of places. However, physical Braille maps are often costly, may not always cover an area of interest with sufficient detail, and might not present up-to-date information. We built a handheld system for representing geographical information called SpaceSense, which includes custom spatial tactile feedback hardware-multiple vibration motors attached to different locations on a mobile touch-screen device. It offers high-level information about the distance and direction towards a destination and bookmarked places through vibrotactile feedback to help the user maintain the spatial relationships between these points. SpaceSense also adapts a summarization technique for online user reviews of public and commercial venues. Our user study shows that participants could build and maintain the spatial relationships between places on a map more accurately with SpaceSense compared to a system without spatial tactile feedback. They pointed specifically to having spatial tactile feedback as the contributing factor in successfully building and maintaining their mental map.

People often complete tasks and to-dos not only for themselves but also for others in their household. In this work, we examine how household members share and accomplish errands both individually and together. We conducted a three-week diary study with eight households to understand the types of errands that family members and roommates share with each other. We explore their motivations for offering and requesting help to complete their errands and the variety of methods for doing so. Our findings reveal when participants sometimes face challenges completing their errands, and how household members request and receive help. We learned that the cooperative performance of errands is typically dependent on household members' location, availability, and capability. Using these findings, we discuss design opportunities for cooperative errands sharing systems that can assist households.

Mobile and handheld devices have become platforms to support remote collaboration. But, their small form-factor may impact the effectiveness of the visual feedback channel often used to help users maintain an awareness of their partner's activities during synchronous collaborative tasks. We investigated how visual and tactile feedback affects collaboration on mobile devices, with emphasis on spatial coordination in a shared workspace. From two user studies, our results highlight different benefits of each feedback channel in collaborative handheld systems. Visual feedback can provide precise spatial information for collaborators, but degrades collaboration when the feedback is occluded, and sometimes can distract the user's attention. Spatial tactile feedback can reduce the overload of information in visual space and gently guides the user's attention to an area of interest. Our results also show that visual and tactile feedback can complement each other, and systems using both feedback channels can support better spatial coordination than systems using only one form of feedback.

Smell can be used to infer quite a bit of context about environments. Previous research primarily has shown that gas sensors can be used to discriminate accurately between odors when used in testing chambers. However, potential real-world applications require these sensors to perform an analysis in uncontrolled environments, which can be challenging. In this poster, we present our gas sensor system, called uSmell, to address these challenges. This system has the potential to improve context-aware applications, such as lifelogging and assisted living.

Accurate activity recognition enables the development of a variety of ubiquitous computing applications, such as context-aware systems, lifelogging, and personal health systems. Wearable sensing technologies can be used to gather data for activity recognition without requiring sensors to be installed in the infrastructure. However, the user may need to wear multiple sensors for accurate recognition of a larger number of different activities. We developed a wearable acoustic sensor, called BodyScope, to record the sounds produced in the user's throat area and classify them into user activities, such as eating, drinking, speaking, laughing, and coughing. The F-measure of the Support Vector Machine classification of 12 activities using only our BodyScope sensor was 79.5%. We also conducted a small-scale in-the-wild study, and found that BodyScope was able to identify four activities (eating, drinking, speaking, and laughing) at 71.5% accuracy.

A rich cognitive map of a space can enhance the individual's experience within the space. However, cognitive maps develop gradually through repeated experience; and because of this, on-demand mobile search services (e.g., Google Maps, Yelp) are often used to compensate for missing knowledge. In this work, we developed and evaluated a context-aware place discovery application called Opportunities Exist to assist in the acquisition of spatial knowledge and meaning. The application differs from traditional search in that places are discovered using an activity (e.g., drink coffee, sit in the sun) and the discovery process runs continuously, maintaining a history of places the user can perform her activities as she goes about her day. We conducted a 4-week deployment in two North American cities. The results show that users were able to discover new places to perform their activities in familiar spaces and learned to associate new activities with familiar places. In addition, participants leveraged the application to perform activities opportunistically, and used continuous place discovery as an opportunistic reminder of routines they wanted to break out of or resume.

The busyness of everyday life means that those with casual interest in additional learning opportunities are often unable to schedule regular time and effort for studying. In this paper, we explore how to augment information technologies that people use on a daily basis to create micro-learning opportunities. In particular, we examine how a person's existing Web browsing experience-with first language Web pages-can be augmented to teach them second language vocabulary. We present a prototype, ALOE, which runs inside the Firefox Web browser and dynamically augments Web pages by replacing a selected set of English words with their foreign translations. The foreign translations are embedded in the rich context of a Web page's existing English text to promote incidental learning and guessing from context of the translated words. Through a two month user evaluation of ALOE, we found that most participants were able to learn an average of 50 new French vocabulary words.

Many people read online reviews written by other users to learn more about a product or venue. However, the overwhelming amount of user-generated reviews and variance in length, detail and quality across the reviews make it difficult to glean useful information. In this paper, we present the iterative design of our system, called Review Spotlight. It provides a brief overview of reviews using adjective-noun word pairs, and allows the user to quickly explore the reviews in greater detail. Through a laboratory user study which required participants to perform decision making tasks, we showed that participants could form detailed impressions about restaurants and decide between two options significantly faster with Review Spotlight than with traditional review webpages.

Many people read online reviews written by other users to learn more about a product or venue. However, the overwhelming amount of user-generated reviews and variance in length, detail and quality across the reviews make it difficult to glean useful information. In this paper, we present a summarization system called Review Spotlight. It provides a brief overview of reviews by using adjective-noun word pairs extracted from the review text. The system also allows the user to click any word pair to read the original sentences from which the word pair was extracted. We present our system implementation as a Google Chrome browser extension, and an evaluation on how two word pair scoring methods (TF and TF-IDF) affect the identification of useful word pairs. 1

Current soft QWERTY keyboards often consume a large portion of the screen space on portable touchscreens. This space consumption can diminish the overall user experi-ence on these devices. In this paper, we present the 1Line keyboard, a soft QWERTY keyboard that is 140 pixels tall (in landscape mode) and 40% of the height of the native iPad QWERTY keyboard. Our keyboard condenses the three rows of keys in the normal QWERTY layout into a single line with eight keys. The sizing of the eight keys is based on users' mental layout of a QWERTY keyboard on an iPad. The system disambiguates the word the user types based on the sequence of keys pressed. The user can use flick gestures to perform backspace and enter, and tap on the bezel below the keyboard to input a space. Through an evaluation, we show that participants are able to quickly learn how to use the 1Line keyboard and type at a rate of over 30 WPM after just five 20-minute typing sessions. Using a keystroke level model, we predict the peak expert text entry rate with the 1Line keyboard to be 66--68 WPM.

Accessing the advanced functions of a mobile phone is not a trivial task for users with visual impairments. They rely on screen readers and voice commands to discover and execute functions. In mobile situations, however, screen readers are not ideal because users may depend on their hearing for safety, and voice commands are difficult for a system to recognize in noisy environments. In this paper, we extend Virtual Shelves--an interaction technique that leverages proprioception to access application shortcuts--for visually impaired users. We measured the directional accuracy of visually impaired participants and found that they were less accurate than people with vision. We then built a functional prototype that uses an accelerometer and a gyroscope to sense its position and orientation. Finally, we evaluated the interaction and prototype by allowing participants to customize the placement of seven shortcuts within 15 regions. Participants were able to access shortcuts in their personal layout with 88.3% accuracy in an average of 1.74 seconds.

Posing a question to an online question and answer community does not guarantee a response. Significant prior work has explored and identified members' motivations for contributing to communities of collective action (e.g., Yahoo! Answers); in contrast it is not well understood why members choose to not answer a question they have already read. To explore this issue, we surveyed 135 active members of Yahoo! Answers. We show that top and regular contributors experience the same reasons to not answer a question: subject nature and composition of the question; perception of how the questioner will receive, interpret and react to their response; and a belief that their response will lose its meaning and get lost in the crowd if too many responses have already been given. Informed by our results, we discuss opportunities to improve the efficacy of the question and answer process, and to encourage greater contributions through improved design.

Community-authored content, such as location specific reviews, offers a wealth of information about virtually every imaginable location today. In this work, we process Yelp's community-authored reviews to identify a set of potential activities that are supported by the location reviewed. Using 14 test locations we show that the majority of the 40 most common results per location (determined by verb-noun pair frequency) are actual activities supported by their respective locations, achieving a mean precision of up to 79.3%. Although the number of reviews authored for a location has a strong influence on precision, we are able to achieve a precision up to 29.5% when processing only the first 50 reviews, increasing to 45.7% and 57.3% for the first 100 and 200 reviews, respectively. In addition, we present two context-aware services that leverage location-based activity information on a city scale that is accessible through a Web service we developed supporting multiple cities in North America.

Battery interfaces provide important feedback about how much time users can continue using their mobile devices. Based on this information, they may develop mental models of the types of activities, tasks, and applications they can use before needing to recharge. Many of today's battery interfaces tend to report energy in coarse granularities or are highly inaccurate. As a result, users may find it difficult to depend on the estimates given. We conducted a survey with 104 participants to understand how users interact with various mobile battery interfaces. Based on the survey results, we designed and prototyped a task-centered battery interface on a mobile device that shows more accurate information about how long individual and combinations of tasks with several applications can be performed. Our pilot study of eight users demonstrated that fine-grained information separated by tasks can help users be more effective with and increase their understanding of their device's battery usage.

Mapping applications on mobile devices have gained widespread popularity as a means for enhancing user mobility and ability to explore new locations and venues. Visually impaired users currently rely on computer text-to-speech or human-spoken descriptions of maps and indoor spaces. Unfortunately, speech-based descriptions are limited in their ability to succinctly convey complex layouts or spacial positioning.

Home networks are common but notoriously difficult to setup and maintain. The difficulty users experience in setting up and maintaining their home network is problematic because of the numerous security threats that can exploit poorly configured and maintained network security. Because there is little empirical data to characterize the usability problems associated with the adoption of wireless network security, we surveyed primary caretakers and users of 20 home networks, examining their perceptions and usage of the security features available to them. We found that users did not understand the difference between access control lists and encryption, and that devices fail to properly notify users of weak security configuration choices. To address these issues, we designed and evaluated a novel wireless router configuration wizard that encouraged strong security choices by improving the network configuration steps. We found that security choices made by users of our wizard resulted in stronger security practices when compared to the wizard from a leading equipment manufacturer.

Visually demanding interfaces on a mobile phone can diminish the user experience by monopolizing the user's attention when they are focusing on another task and impede accessibility for visually impaired users. Because mobile devices are often located in pockets when users are mobile, explicit foot movements can be defined as eyes-and-hands-free input gestures for interacting with the device. In this work, we study the human capability associated with performing foot-based interactions which involve lifting and rotation of the foot when pivoting on the toe and heel. Building upon these results, we then developed a system to learn and recognize foot gestures using a single commodity mobile phone placed in the user's pocket or in a holster on their hip. Our system uses acceleration data recorded by a built-in accelerometer on the mobile device and a machine learning approach to recognizing gestures. Through a lab study, we demonstrate that our system can classify ten different foot gestures at approximately 86% accuracy.

In co-located software development, diagramming practices, such as sketching ideas out with a pen and paper, support the creative process and allow designers to shape, analyze, and communicate their ideas. This study focuses on the diagramming practices used in the design of Open Source Software (OSS), where the norm is highly distributed group work. In OSS, text-based communication (e.g., mailing lists) dominates, and sketching and drawing diagrams collaboratively remains difficult due to the barriers imposed by distance and technology. Previous studies have examined these practices and barriers in the context of individual projects. To understand how contributors across OSS projects use diagrams in design-related activities, we conducted a survey of 230 contributors from 40 different OSS projects, and interviewed eight participants. Our results show that although contributors understand the advantages of using diagrams for design-related activities, diagrams are infrequently used in OSS. This motivated us to examine how and why diagramming occurs, and the factors that prevent widespread use in OSS. Finally, we propose new ideas for supporting design activities in OSS projects.

When gathering feedback about an envisioned system, prototypes communicate design ideas to user groups. However, it is unclear how user responses are affected by prototype format. We conducted a 2x2 quasi-experiment (video /storyboard format x older and younger user groups) to test for an interaction between prototype format and user group. We found a significant interaction between prototype format and responses across user groups. Our results indicate that differences in user responses can be misinterpreted as the result of user group characteristics. We advise using multiple prototype formats to counteract a 'media effect'. Alternatively, we advise using storyboards for a smaller 'media effect'.

Exercising is often a social activity performed with other people, yet finding compatible exercise partners is difficult in practice. To gain a better understanding of the social requirements involved with forming exercise groups, we conducted a two-phased exploratory study involving an online web questionnaire with 96 respondents and two focus groups. Our results highlight various aspects of collaborating with exercise partners, but also indicate the limited utility of currently available systems to support such collaborations. We discuss implications for collaborative technologies supporting exercise group formation.

Some of the most interesting differences between Open Source Software (OSS) development and commercial co-located software development lie in the communication and collaboration practices of these two groups of developers. One interesting practice is that of diagramming. Though well studied and important in many aspects of co-located software development (including communication and collaboration among developers), its role in OSS development has not been thoroughly studied. In this paper, we report our investigation on how and why Ubuntu contributors use diagrams in their work. Our study shows that diagrams are not actively used in many scenarios where they commonly would in co-located software development efforts. We describe differences in the use and practices of diagramming, their possible reasons, and present design considerations for potential systems aimed at better supporting diagram use in OSS development.

Large information displays are common in public and semi-public spaces but still require rapid and lightweight ways for users to interact with them. We present BlueTone, a framework for developing large display applications which will interpret and react to dual-tone multi-frequency sounds transmitted from mobile phones paired with the display using the Bluetooth headset profile. BlueTone enables text entry, cursor manipulation and menu selection without requiring the installation of any special software on a user's mobile phone.

In this paper, we present a study of responses to the idea of being recorded by a ubicomp recording technology called SenseCam. This study focused on real-life situations in two North American and two European locations. We present the findings of this study and their implications, specifically how those who might be recorded perceive and react to SenseCam. We describe what system parameters, social processes, and policies are required to meet the needs of both the primary users and these secondary stakeholders and how being situated within a particular locale can influence responses. Our results indicate that people would tolerate potential incursions from SenseCam for particular purposes. Furthermore, they would typically prefer to be informed about and to consent to recording as well as to grant permission before any data is shared. These preferences, however, are unlikely to instigate a request for deletion or other action on their part. These results inform future design of recording technologies like SenseCam and provide a broader understanding of how ubicomp technologies might be taken up across different cultural and political regions.

Triggering shortcuts or actions on a mobile device often requires a long sequence of key presses. Because the functions of buttons are highly dependent on the current application's context, users are required to look at the display during interaction, even in many mobile situations when eyes-free interactions may be preferable. We present Virtual Shelves, a technique to trigger programmable shortcuts that leverages the user's spatial awareness and kinesthetic memory. With Virtual Shelves, the user triggers shortcuts by orienting a spatially-aware mobile device within the circular hemisphere in front of her. This space is segmented into definable and selectable regions along the phi and theta planes. We show that users can accurately point to 7 regions on the theta and 4 regions on the phi plane using only their kinesthetic memory. Building upon these results, we then evaluate a proof-of-concept prototype of the Virtual Shelves using a Nokia N93. The results show that Virtual Shelves is faster than the N93's native interface for common mobile phone tasks.

One of the challenges with using mobile touch-screen devices is that they do not provide tactile feedback to the user. Thus, the user is required to look at the screen to interact with these devices. In this paper, we present SemFeel, a tactile feedback system which informs the user about the presence of an object where she touches on the screen and can offer additional semantic information about that item. Through multiple vibration motors that we attached to the backside of a mobile touch-screen device, SemFeel can generate different patterns of vibration, such as ones that flow from right to left or from top to bottom, to help the user interact with a mobile device. Through two user studies, we show that users can distinguish ten different patterns, including linear patterns and a circular pattern, at approximately 90% accuracy, and that SemFeel supports accurate eyes-free interactions.

Handheld computers have untapped potential to improve HIV/AIDS programs in sub-Saharan Africa, particularly in the collection of survey data. We conducted an experiment in three neighborhoods of Luanda, Angola to assess the impact of the technology on people's comfort and willingness to disclose sensitive personal information, such as sexual behavior. Participants were asked about their HIV/AIDS-related knowledge, attitudes, and practices by local interviewers using either handheld computers or paper surveys. T-tests showed no differences between participants' self-reported comfort across handheld and paper conditions. However, participants in the handheld condition were more likely to give socially desirable responses to the sexual behavior questions than participants in the paper condition. These results suggest that using handheld computers in data collection in sub-Saharan Africa may lead to biased reports of HIV/AIDS-related risk behaviors.

Identifying the function of problem behavior can lead to the development of more effective interventions. One way to identify the function is through functional behavior assessment (FBA). Teachers conduct FBA in schools. However, the task load of recording the data manually is high, and the challenge of accurately identifying antecedents and consequences is significant while interacting with students. These issues often result in imperfect information capture. CareLog allows teachers more easily to conduct FBAs and enhances the capture of relevant information. In this paper, we describe the design process that led to five design principles that governed the development of CareLog. We present results from a five-month, quasi-controlled study aimed at validating those design principles. We reflect on how various constraints imposed by special education settings impact the design and evaluation process for HCI practitioners and researchers.

We present a qualitative study of mobile phone ownership, replacement and disposal practices geared towards identifying design opportunities towards sustainable mobile phone interfaces. Our work investigates how people understand the lifespan of their phones, what factors, such as style, service contracts, and functionality, affect how they attribute value to their phones, and their awareness and actions regarding mobile phone sustainability. Our findings reveal the complexity of the actions and decision-making processes involved in phone ownership and replacement. We use these findings to present open areas for sustainable interaction design and generate seed ideas for designs and services to provoke thought and further exploration towards more sustainable mobile phone interfaces and practices.

A person often has highly context-sensitive information needs that require assistance from individuals in their social network. However, a person's social network is often not broad enough to include the right people in the right situations or circumstances who can satisfy the needs. The ability to satisfy context-sensitive information needs depends on a person's ability to seek the answers from appropriate individuals, who must then provide a response in a timely manner. To gain an understanding of how to better support the sharing of information, we conducted a four-week diary study examining 20 people's perceived daily information needs and sharing desires. We provide a structured framework for understanding the types of information people need and discuss when and how people are able to satisfy their needs. Using these findings, we discuss research and design opportunities for addressing the shortcomings of the existing information sources by connecting information altruists with an audience by leveraging weak ties through situation and circumstance, and providing a timely asynchronous connection to these sources.

In this paper, we explore the use of an interaction technique called sequential segmentation to support target selection for mobile devices. Sequential segmentation iteratively partitions an information space into selectable regions and subsequent sub-regions where each region/sub-region is labeled (1--9) and is mapped to the corresponding key on the mobile device's numeric keypad. We conducted a study comparing the sequential technique to the directional pad for target selection. The results show that the directional pad is significantly faster than sequential for selecting targets that require three or less interactions with the directional pad. However, sequential is significantly faster than the directional pad for targets that require five or more interactions with the directional pad. User feedback shows a preference for sequential and that it is perceived easier to use than the directional pad.

Effective text entry on handheld devices remains a significant problem in the field of mobile computing. On a personal digital assistant (PDA), text entry methods traditionally support input through the motion of a stylus held in the user's dominant hand. In this paper, we present the design of a two-handed software keyboard for a PDA which specifically takes advantage of the thumb in the non-dominant hand. We compare our chorded keyboard design to other stylus-based text entry methods in an evaluation that studies user input in both stationary and mobile settings. Our study shows that users type fastest using the miniqwerty keyboard, and most accurately using our two-handed keyboard. We also discovered a difference in input performance with the mini-qwerty keyboard between stationary and mobile settings. As a user walks, text input speed decreases while error rates and mental workload increases; however, these metrics remain relatively stable in our two-handed technique despite user mobility.

Most pages on the Web are designed for the desktop environment and render poorly on the small screens available on handheld devices. We introduce Reusable End-User Customization (REUC), a technique that lets end users adapt the layout of Web pages by removing, resizing and moving page elements. REUC records the user's customizations and automatically reapplies them on subsequent visits to the same page or to other, similar pages, on the same Web site. We present PageTailor, a REUC prototype based on the Minimo Web browser that runs on Windows Mobile PDAs. We show that users can utilize PageTailor to adapt sophisticated Web sites, such as Amazon, BBC and MSN, for browsing on a PDA. Moreover, the customizations remain effective for up to a year, even as the content of pages is updated, and can be reused across similar pages, limiting the customization effort required to browse a site.

Storyboarding is a common technique in HCI and design for demonstrating system interfaces and contexts of use. Despite its recognized benefits, novice designers still encounter challenges in the creation of storyboards. Furthermore, as computing becomes increasingly integrated into the environment, blurring the distinction between the system and its surrounding context, it is imperative to depict context explicitly in storyboards. In this paper, we present two formative studies designed to uncover the important elements of storyboards. These elements include the use of text, inclusion of people, level of detail, number of panels, and representation of the passage of time. We further present an empirical study to assess the effects of these elements on the understanding and enjoyment of storyboard consumers. Finally, we demonstrate how these guidelines were successfully used in an undergraduate HCI class.

We developed an inquiry technique, which we called "paratype," based on experience prototyping and event-contingent experience sampling, to survey people in real-life situations about ubiquitous computing (ubicomp) technology. We used this tool to probe the opinions of the conversation partners of users of the Personal Audio Loop, a memory aid that can have a strong impact on their privacy. We present the findings of this study and their implications, specifically the need to broaden public awareness of ubicomp applications and the unfitness of traditional data protection guidelines for tackling the privacy issues of many ubicomp applications. We also point out benefits and methodological issues of paratypes and discuss why they are particularly fit for studying certain classes of mobile and ubicomp applications.

Both content driven web authors and application designers may have their attention deviated from their main task when they have to be concerned with the generation of elaborated linking structures. This work aims to demonstrate how a metadata-enhanced web-based open linking service can be exploited towards supporting content driven authors in their tasks. The following results are presented in this paper: (a) the Web Linking Service (WLS), a novel open hypermedia system that stores and exchanges metadata in RDF standard syntax for hypertext structures across the wire and (b) two case studies in which applications offer to their users the ability to create linking structures upon existing contents by making use the WLS service.

In this paper, we describe our work on developing a system to support the personalization of a captured public experience. Specifically, we are interested in providing students with the ability to personalize the capture of the lecture experiences as part of the Classroom 2000 project. We discuss the issues and challenges involved in designing a system that performs live integration of personal streams of information with multiple other streams of information made available to it through an environment designed to capture public information.