Experiments in human-computer interaction (HCI) often evaluate whether a prototype is “better,” but novelty alone can affect users’ judgments and possibly performance. To quantify this effect, we conducted a within-subjects study of 48 participants comparing four pairs of functionally identical prototypes (mice, keyboards, search engines, and AI chatbots). Each pair differed only in cosmetic features and a label marking one as “old” and the other as “new.” Novelty labeling shifted preference: up to 77% favored the version labeled “new.” Subjective ratings for the search engine increased under the “new’’ label by up to 7.1%. For the AI chatbot, ratings were driven by preference, with the preferred version rated up to 11.6% higher than the unpreferred one. Performance differences were modest and emerged for errors (e.g., 9.7% fewer misses with the “new” mouse, up to 7.2% lower error rates with the “new” keyboard). Technology readiness predicted baseline skill and occasionally moderated performance but did not protect judgments from novelty bias. These results show that novelty labeling reframes interpretation and preference more than performance, raising concerns for HCI evaluations relying on participant judgments.

The accessibility of interactive technologies is often evaluated using checklists that are low-level, numerous, and platform-specific. Such checklists are typically used by accessibility experts, leaving everyday designers and developers with little support for assessing their own interfaces. To make accessibility evaluations easier to conduct, we devised a set of nine “ability heuristics” that prompt designers to engage with accessibility throughout the design process. We empirically evaluated these ability heuristics with 37 design students, comparing them to usability heuristics and WCAG. The ability heuristics emphasized the quality of accessibility features compared to the other methods, and surfaced issues that were more broadly dispersed across disability groups. Further, the students found the heuristics were as easy to use as the alternative methods. We argue that the heuristics help to move beyond binary notions of accessibility, pushing designers to consider the quality of features across diverse disabilities and the range of abilities within.

Accessibility checkers are tools in support of accessible app development, and their use is encouraged by accessibility best practices. However, most current checkers evaluate static or mechanically-generated contexts, failing to capture common accessibility errors impacting mobile app functionality. In this work, we define functiona11ity errors as accessibility barriers that only manifest through interaction (i.e., named according to a blend of "functionality" and "accessibility"). We introduce TaskAudit, which comprises three components: a Task Generator that constructs interactive tasks from app screens, a Task Executor that uses agents with a screen reader proxy to perform these tasks, and an Accessibility Analyzer that detects and reports accessibility errors by examining interaction traces. Our evaluation on real-world apps shows that TaskAudit detects 48 functiona11ity errors from 54 app screens, compared to between 4 and 20 with existing checkers. Our analysis demonstrates common error patterns that TaskAudit can detect in addition to those from prior work, including label-functionality mismatch, cluttered navigation, and inappropriate feedback.

Figure 1: With A11yShape, (A) a blind or low-vision (BLV) user can create, interpret, and verify 3-D models through (B) a user interface composed of three parts: Code Editor Panel, AI Assistant Panel, and Model Panel.These panels are linked by a cross-representation highlighting mechanism that connects code, textual descriptions, hierarchical model abstractions, and 3-D visual renderings.The system supports the creation of (C) diverse, customized 3-D models created by BLV users.

From curating the layout of a resume to selecting filters for social media, creating and configuring visual content allows individuals to express identity, communicate intent, and engage socially, yet blind individuals often face significant barriers to such expressive practices.Prior accessibility research primarily addresses functional content configuration, leaving little understanding of blind individuals' expressive visual creation needs.To better understand and support these needs, we conducted a two-stage study: first, we interviewed 10 blind participants to understand their motivations, current practices, and barriers in visual expression, and to ideate on potential visual editing support; second, based on interview insights, we developed an interactive prototype (VizXpress) that provides real-time feedback on visual aesthetics using a vision-language model and supports automated and manual visual editing controls.We used VizXpress as a design probe to further explore accessible design opportunities for visual expression.Our findings highlight many blind users' strong interest in creating visually expressive content, nuanced informational requirements for subjective aesthetics (e.g., color, mood, lighting), and ongoing accessibility challenges with visual creative tools.Grounded in these insights, we propose design implications including richer aesthetic feedback, controlled intelligent editing, and accessible manual editing mechanisms.

Many mobile apps are inaccessible, thereby excluding people from their potential benefits. Existing rule-based accessibility checkers aim to mitigate these failures by identifying errors early during development but are constrained in the types of errors they can detect. We present ScreenAudit, an LLM-powered system designed to traverse mobile app screens, extract metadata and transcripts, and identify screen reader accessibility errors overlooked by existing checkers. We recruited six accessibility experts including one screen reader user to evaluate ScreenAudit's reports across 14 unique app screens. Our findings indicate that ScreenAudit achieves an average coverage of 69.2%, compared to only 31.3% with a widely-used accessibility checker. Expert feedback indicated that ScreenAudit delivered higher-quality feedback and addressed more aspects of screen reader accessibility compared to existing checkers, and that ScreenAudit would benefit app developers in real-world settings.

We introduce ArtInsight, a novel AI-powered system to facilitate deeper engagement with child-created artwork in mixed visual-ability families. ArtInsight leverages large language models (LLMs) to craft a respectful and thorough initial description of a child's artwork, and provides: creative AI-generated descriptions for a vivid overview, audio recording to capture the child's own description of their artwork, and a set of AI-generated questions to facilitate discussion between blind or low-vision (BLV) family members and their children. Alongside ArtInsight, we also contribute a new rubric to score AI-generated descriptions of child-created artwork and an assessment of state-of-the-art LLMs. We evaluated ArtInsight with five groups of BLV family members and their children, and as a case study with one BLV child therapist. Our findings highlight a preference for ArtInsight's longer, artistically-tailored descriptions over those generated by existing BLV AI tools. Participants highlighted the creative description and audio recording components as most beneficial, with the former helping ``bring a picture to life'' and the latter centering the child's narrative to generate context-aware AI responses. Our findings reveal different ways that AI can be used to support art engagement, including before, during, and after interaction with the child artist, as well as expectations that BLV adults and their sighted children have about AI-powered tools.

We introduce ChartA11y, an app developed to enable accessible 2-D visualizations on smartphones for blind users through a participatory and iterative design process involving 13 sessions with two blind partners. We also present a design journey for making accessible touch experiences that go beyond simple auditory feedback, incorporating multimodal interactions and multisensory data representations. Together, ChartA11y aimed at providing direct chart accessing and comprehensive chart understanding by applying a two-mode setting: a semantic navigation framework mode and a direct touch mapping mode. By re-designing traditional touch-to-audio interactions, ChartA11y also extends to accessible scatter plots, addressing the under-explored challenges posed by their non-linear data distribution. Our main contributions encompass the detailed participatory design process and the resulting system, ChartA11y, offering a novel approach for blind users to access visualizations on their smartphones.

We present two studies exploring how blind or low-vision (BLV) family members engage with their sighted children’s artwork, strategies to support understanding and interpretation, and the potential role of technology, such as AI, therein. Our first study involved 14 BLV individuals, and the second included five groups of BLV individuals with their children. Through semi-structured interviews with AI descriptions of children’s artwork and multi-sensory design probes, we found that BLV family members value artwork engagement as a bonding opportunity, preferring the child’s storytelling and interpretation over other nonvisual representations. Additionally, despite some inaccuracies, BLV family members felt that AI-generated descriptions could facilitate dialogue with their children and aid self-guided art discovery. We close with specific design considerations for supporting artwork engagement in mixed visual-ability families, including enabling artwork access through various methods, supporting children’s corrections of AI output, and distinctions in context vs. content and interpretation vs. description of children’s artwork.

Ability-based design (Wobbrock et al. 2011, 2018) offers conceptual guidance for its use in designing accessible systems, but the construct of “ability” itself—a crucial notion for ability-based design—is surprisingly elusive and absent from extant accounts. Different disciplines offer disparate notions regarding definitions and measures of “ability,” but ability-based design has yet to avail itself of these notions in its operationalization. To address this gap, this work reviews literature that quantifies motor abilities, provides guidance to distill metrics for human-computer interaction, and conceptualizes how motor ability should be quantified for ability-based design. We offer a three-dimensional framework composed of the user, context, and task, and we locate various metrics for ability to be used when implementing ability-based designs. We support this new conceptualization with example personas that occupy this three-dimensional space. This work can inform those using ability-based design to create systems that are responsive to users’ abilities.

We sought to understand and reduce the challenges creators face with making their data visualizations accessible. Specifically, we administered a formative survey of 57 creators to comprehend their challenges, perceived importance, knowledge, and prioritization of data visualization accessibility. Participants identified five interventions to minimize their challenges: Workshops, Emulators, Evaluators, Feedback Collectors, and Multi-Modal Automated Tools. Additionally, we report specifications and recommendations from 12 visualization creators for effective versions of each intervention, gathered via semi-structured interviews. Utilizing our findings, such as a “mini-survey” format that is effective for collecting accessibility-related feedback from screen-reader users, we implemented and integrated these interventions into VoxLens (Sharif et al., 2022). We assessed our enhancements through a task-based user study with 10 visualization creators, finding 44%, 17%, and 12% improvements in their understanding of screen-reader users’ challenges with data visualizations, knowledge of visualization accessibility, and perceived usefulness of the enhanced VoxLens, respectively.

Touchpads are used widely to interact with computers, yet they provide minimal utility for screen-reader users. We explore the utility of touchpads as input devices for screen-reader users through the development and preliminary evaluation of Touchpad Mapper. This system maps digital content (i.e., images and videos) to the physical coordinates of a touchpad. We examined two usage scenarios: (1) identification of objects and their relative positioning in an image and (2) controlling a video seek bar and slider with rewinding and fast-forwarding features. We conducted task-based semi-structured interviews with two screen-reader users to assess our system’s performance. The participants reported positive experiences, highlighting that they extracted information faster using our system than the conventional keyboard-only interaction.

Despite significant progress in the capabilities of mobile devices and applications, most apps remain oblivious to their users' abilities. To enable apps to respond to users' situated abilities, we created the Ability-Based Design Mobile Toolkit (ABD-MT). ABD-MT integrates with an app's user input and sensors to observe a user's touches, gestures, physical activities, and attention at runtime, to measure and model these abilities, and to adapt interfaces accordingly. Conceptually, ABD-MT enables developers to engage with a user's "ability profile,'' which is built up over time and inspectable through our API. As validation, we created example apps to demonstrate ABD-MT, enabling ability-aware functionality in 91.5% fewer lines of code compared to not using our toolkit. Further, in a study with 11 Android developers, we showed that ABD-MT is easy to learn and use, is welcomed for future use, and is applicable to a variety of end-user scenarios.

Presentation software like Microsoft PowerPoint and Google Slides remains largely inaccessible for blind users because screen readers are not well suited to 2-D “artboards” that contain different objects in arbitrary arrangements lacking any inherent reading order. To investigate this problem, prior work by Zhang & Wobbrock (2023) developed multimodal interaction techniques in a prototype system called A11yBoard, but their system was limited to a single artboard in a self-contained prototype and was unable to support real-world use. In this work, we present a major extension of A11yBoard that expands upon its initial interaction techniques, addresses numerous real-world issues, and makes it deployable with Google Slides. We describe the new features developed for A11yBoard for Google Slides along with our participatory design process with a blind co-author. We also present two case studies based on real-world deployments showing that participants were able to independently complete slide reading and authoring tasks that were not possible without sighted assistance previously. We conclude with several design guidelines for making accessible digital content creation tools.

Although virtual reality (VR) is becoming increasingly popular and many interaction techniques for navigating virtual environments, known as “locomotion techniques,” exist, data regarding the accessibility of locomotion techniques for people with upper-body motor impairments do not exist, making it difficult to understand which locomotion techniques work well for users and why. To address this gap, we conducted a study with 19 participants with upper-body motor impairments who completed a navigation task in VR using six seated locomotion techniques. We collected task performance data and elicited participant feedback using questionnaires and interviews. We found that participants performed similarly well with three techniques that required one controller input, little to no upper-body movement, and had a low perceived workload: Teleport, Astral Body, and Sliding Looking. However, Teleport was consistently favored in interview responses and could be considered the best technique for this group of participants. On the other hand, participants performed similarly poorly with three techniques that required arm, head, and torso movement and also rated these techniques as having a high workload: Chicken Acceleration, Grab and Pull, and Throw Teleport. However, participants did not necessarily prefer or want to use the techniques with which they performed best or had the lowest perceived workloads. Factors such as enjoyment, exercise, and presence sometimes outweighed accessibility. This finding suggests that accessibility alone should not override all other considerations when designing or recommending locomotion techniques to people with upper-body motor impairments, and that users with disabilities should have a range of accessible locomotion techniques to choose from based on their preferences.

Always-on, upper-body input from sensors like accelerometers, infrared cameras, and electromyography hold promise to enable accessible gesture input for people with upper-body motor impairments. When these sensors are distributed across the person's body, they can enable the use of varied body parts and gestures for device interaction. Personalized upper-body gestures that enable input from diverse body parts including the head, neck, shoulders, arms, hands and fingers and match the abilities of each user, could be useful for ensuring that gesture systems are accessible. In this work, we characterize the personalized gesture sets designed by 25 participants with upper-body motor impairments and develop design recommendations for upper-body personalized gesture interfaces. We found that the personalized gesture sets that participants designed were highly ability-specific. Even within a specific type of disability, there were significant differences in what muscles participants used to perform upper-body gestures, with some pre-dominantly using shoulder and upper-arm muscles, and others solely using their finger muscles. Eight percent of gestures that participants designed were with their head, neck, and shoulders, rather than their hands and fingers, demonstrating the importance of tracking the whole upper-body. To combat fatigue, participants performed 51% of gestures with their hands resting on or barely coming off of their armrest, highlighting the importance of using sensing mechanisms that are agnostic to the location and orientation of the body. Lastly, participants activated their muscles but did not visibly move during 10% of the gestures, demonstrating the need for using sensors that can sense muscle activations without movement. Both inertial measurement unit (IMU) and electromyography (EMG) wearable sensors proved to be promising sensors to differentiate between personalized gestures. Personalized upper-body gesture interfaces that take advantage of each person's abilities are critical for enabling accessible upper-body gestures for people with upper-body motor impairments.

Digital artboards, which hold objects rather than pixels (e.g., Microsoft PowerPoint and Google Slides), remain largely inaccessible for blind and low-vision (BLV) users. Building on prior findings about the experiences of BLV users with digital artboards, we present a novel tool called A11yBoard, an interactive multimodal system that makes interpreting and authoring digital artboards accessible. A11yBoard combines a web-based drawing canvas paired with a mobile touch screen device such as a tablet. The mobile device displays the same canvas and enables risk-free spatial exploration of the artboard via touch and gesture. Speech recognition, non-speech audio, and keyboard-based commands are also used for input and output. Through a series of pilot studies and formal task-based user studies with BLV participants, we show that A11yBoard provides (1) intuitive spatial reasoning about two-dimensional objects, (2) multimodal access to objects’ properties and relationships, and (3) eyes-free creating and editing of objects to establish their desired properties and positions.

Measures of human performance for touch-based systems have focused mainly on overall metrics like touch accuracy and target acquisition speed. But touches are not atomic—they unfold over time and space, especially for users with limited fine motor function, for whom it can be difficult to perform quick, accurate touches. To gain insight into what happens during a touch, we offer 15 target-agnostic touch metrics, most of which have not been mathematically formalized in the literature. They are touch direction, variability, drift, duration, extent, absolute/signed area change, area variability, area deviation, area extent, absolute/signed angle change, angle variability, angle deviation, and angle extent. These metrics regard a touch as a time series of ovals instead of a mere (x, y) coordinate. We provide mathematical definitions and visual depictions of our metrics, and consider policies for calculating our metrics when multiple fingers perform coincident touches. To exercise our metrics, we collected touch data from 27 participants, 15 of whom reported having limited fine motor function. Our results show that our metrics effectively characterize touch behaviors including fine-motor challenges. Our metrics can be useful for both understanding users and for evaluating touch-based systems to inform their design.

Prior work has studied the interaction experiences of screen-reader users with simple online data visualizations (e.g., bar charts, line graphs, scatter plots), highlighting the disenfranchisement of screen-reader users in accessing information from these visualizations. However, the interactions of screen-reader users with online geospatial data visualizations, commonly used by visualization creators to represent geospatial data (e.g., COVID-19 cases per US state), remain unexplored. In this work, we study the interactions of and information extraction by screen-reader users from online geospatial data visualizations. Specifically, we conducted a user study with 12 screen-reader users to understand the information they seek from online geospatial data visualizations and the questions they ask to extract that information. We utilized our findings to generate a taxonomy of information sought from our participants’ interactions. Additionally, we extended the functionalities of VoxLens—an open-source multi-modal solution that improves data visualization accessibility—to enable screen-reader users to extract information from online geospatial data visualizations.

We present the first large-scale longitudinal analysis of missing label accessibility failures in Android apps. We developed a crawler and collected monthly snapshots of 312 apps over 16 months. We use this unique dataset in empirical examinations of accessibility not possible in prior datasets. Key large-scale findings include missing label failures in 55.6% of unique image-based elements, longitudinal improvement in ImageButton elements but not in more prevalent ImageView elements, that 8.8% of unique screens are unreachable without navigating at least one missing label failure, that app failure rate does not improve with number of downloads, and that effective labeling is neither limited to nor guaranteed by large software organizations. We then examine longitudinal data in individual apps, presenting illustrative examples of accessibility impacts of systematic improvements, incomplete improvements, interface redesigns, and accessibility regressions. We discuss these findings and potential opportunities for tools and practices to improve label-based accessibility.

JavaScript visualization libraries are widely used to create online data visualizations but provide limited access to their information for screen-reader users. Building on prior findings about the experiences of screen-reader users with online data visualizations, we present VoxLens, an open-source JavaScript plug-in that—with a single line of code—improves the accessibility of online data visualizations for screen-reader users using a multi-modal approach. Specifically, VoxLens enables screen-reader users to obtain a holistic summary of presented information, play sonified versions of the data, and interact with visualizations in a “drill-down” manner using voice-activated commands. Through task-based experiments with 21 screen-reader users, we show that VoxLens improves the accuracy of information extraction and interaction time by 122% and 36%, respectively, over existing conventional interaction with online data visualizations. Our interviews with screen-reader users suggest that VoxLens is a “game-changer” in making online data visualizations accessible to screen-reader users, saving them time and effort.

Animated GIF images have become prevalent in internet culture, often used to express richer and more nuanced meanings than static images. But animated GIFs often lack adequate alternative text descriptions, and it is challenging to generate such descriptions automatically, resulting in inaccessible GIFs for blind or low-vision (BLV) users. To improve the accessibility of animated GIFs for BLV users, we provide a system called Ga11y (pronounced “galley”), for creating GIF annotations. Ga11y combines the power of machine intelligence and crowdsourcing and has three components: an Android client for submitting annotation requests, a backend server and database, and a web interface where volunteers can respond to annotation requests. We evaluated three human annotation interfaces and employ the one that yielded the best annotation quality. We also conducted a multi-stage evaluation with 12 BLV participants from the United States and China, receiving positive feedback.

We present a QWERTY-based text entry system, TypeAnywhere, for use in off-desktop computing environments. Using a wearable device that can detect finger taps, users can leverage their touch-typing skills from physical keyboards to perform text entry on any surface. TypeAnywhere decodes typing sequences based only on finger-tap sequences without relying on tap locations. To achieve optimal decoding performance, we trained a neural language model and achieved a 1.6% character error rate (CER) in an offline evaluation, compared to a 5.3% CER from a traditional n-gram language model. Our user study showed that participants achieved an average performance of 70.6 WPM, or 80.4% of their physical keyboard speed, and 1.50% CER after 2.5 hours of practice over five days on a table surface. They also achieved 43.9 WPM and 1.37% CER when typing on their laps. Our results demonstrate the strong potential of QWERTY typing as a ubiquitous text entry solution.

We clarify fundamental aspects of end-user elicitation, enabling such studies to be run and analyzed with confidence, correctness, and scientific rigor. To this end, our contributions are multifold. We introduce a formal model of end-user elicitation in HCI and identify three types of agreement analysis:expert,codebook, andcomputer. We show that agreement is a mathematicaltolerance relationgenerating a tolerance space over the set of elicited proposals. We review current measures of agreement and show that all can be computed from anagreement graph. In response to recent criticisms, we show that chance agreement represents an issue solely for inter-rater reliability studies and not for end-user elicitation, where it is opposed bychance disagreement. We conduct extensive simulations of 16 statistical tests for agreement rates, and report Type I errors and power. Based on our findings, we provide recommendations for practitioners and introduce a five-level hierarchy for elicitation studies.

Current performance measures with touch-based systems usually focus on overall performance, such as touch accuracy and target acquisition speed. But a touch is not an atomic event; it is a process that unfolds over time, and this process can be characterized to gain insight into users’ touch behaviors. To this end, our work proposes 13 target-agnostic touch performance metrics to characterize what happens during a touch. These metrics are: touch direction, variability, drift, duration, extent, absolute/signed area change, area variability, area deviation, absolute/signed angle change, angle variability, and angle deviation. Unlike traditional touch performance measures that treat a touch as a single (x, y) coordinate, we regard a touch as a time series of ovals that occur from finger-down to finger-up. We provide a mathematical formula and intuitive description for each metric we propose. To evaluate our metrics, we run an analysis on a publicly available dataset containing touch inputs by people with and without limited fine motor function, finding our metrics helpful in characterizing different fine motor control challenges. Our metrics can be useful to designers and evaluators of touch-based systems, particularly when making touch screens accessible to all forms of touch.

Online data visualizations are widely used to communicate information from simple statistics to complex phenomena, supporting people in gaining important insights from data. However, due to the defining visual nature of data visualizations, extracting information from visualizations can be difficult or impossible for screen-reader users. To assess screen-reader users’ challenges with online data visualizations, we conducted two empirical studies: (1) A qualitative study with nine screen-reader users, and (2) a quantitative study with 36 screen-reader and 36 non-screen-reader users. Our results show that due to the inaccessibility of online data visualizations, screen-reader users extract information 61.48% less accurately and spend 210.96% more time interacting with online data visualizations compared to non-screen-reader users. Additionally, our findings show that online data visualizations are commonly indiscoverable to screen readers. In visualizations that are discoverable and comprehensible, screen-reader users suggested tabular and textual representation of data as techniques to improve the accessibility of online visualizations. Taken together, our results provide empirical evidence of the inequalities screen-readers users face in their interaction with online data visualizations.

Priming is used as a way of increasing the diversity of proposals in end-user elicitation studies, but priming has not been investigated thoroughly in this context. We conduct a distributed end-user elicitation study with 167 participants, which had three priming groups: a no-priming control group, sci-fi priming, and a creative mindset group. We evaluated the gestures proposed by these groups in a distributed learnability and memorability study with 18 participants. We found that the user-elicited gestures from the sci-fi group were significantly faster to learn, requiring an average of 1.22 viewings to learn compared to 1.60 viewings required to learn the control gestures, and 1.56 viewings to learn the gestures elicited from the creative mindset group. In addition, both primed gesture groups had higher memorability with 80% of the sci-fi-primed gestures and 73% of the creative mindset group gestures were recalled correctly after one week without practice compared to 43% of the control group gestures.

Two-dimensional canvases are the core components of many digital productivity and creativity tools, with “artboards” containing objects rather than pixels. Unfortunately, the contents of artboards remain largely inaccessible to blind users relying on screen-readers, but the precise problems are not well understood. This study sought to understand how blind screen-reader users interact with artboards. Specifically, we conducted contextual interviews, observations, and task-based usability studies with 15 blind participants to understand their experiences of artboards found in Microsoft PowerPoint, Apple Keynote, and Google Slides. Participants expressed that the inaccessibility of these artboards contributes to significant educational and professional barriers. We found that the key problems faced were: (1) high cognitive loads from a lack of feedback about artboard contents and object state; (2) difficulty determining relationships among artboard objects; and (3) constant uncertainty about whether object manipulations were successful. We offer design remedies that improve feedback for object state, relationships, and manipulations.

Keyboard-based emoji entry can be challenging for people with visual impairments: users have to sequentially navigate emoji lists using screen readers to find their desired emojis, which is a slow and tedious process. In this work, we explore the design and benefits of emoji entry with speech input, a popular text entry method among people with visual impairments. After conducting interviews to understand blind or low vision (BLV) users’ current emoji input experiences, we developed Voicemoji, which (1) outputs relevant emojis in response to voice commands, and (2) provides context-sensitive emoji suggestions through speech output. We also conducted a multi-stage evaluation study with six BLV participants from the United States and six BLV participants from China, finding that Voicemoji significantly reduced entry time by 91.2% and was preferred by all participants over the Apple iOS keyboard. Based on our findings, we present Voicemoji as a feasible solution for voice-based emoji entry.

Data from multifactor HCI experiments often violates the assumptions of parametric tests (i.e., nonconforming data). The Aligned Rank Transform (ART) has become a popular nonparametric analysis in HCI that can find main and interaction effects in nonconforming data, but leads to incorrect results when used to conduct post hoc contrast tests. We created a new algorithm called ART-C for conducting contrast tests within the ART paradigm and validated it on 72,000 synthetic data sets. Our results indicate that ART-C does not inflate Type I error rates, unlike contrasts based on ART, and that ART-C has more statistical power than a t-test, Mann-Whitney U test, Wilcoxon signed-rank test, and ART. We also extended an open-source tool called ARTool with our ART-C algorithm for both Windows and R. Our validation had some limitations (e.g., only six distribution types, no mixed factorial designs, no random slopes), and data drawn from Cauchy distributions should not be analyzed with ART-C.

Gesture elicitation studies represent a popular and resourceful method in HCI to inform the design of intuitive gesture commands, reflective of end-users' behavior, for controlling all kinds of interactive devices, applications, and systems. In the last ten years, an impressive body of work has been published on this topic, disseminating useful design knowledge regarding users' preferences for finger, hand, wrist, arm, head, leg, foot, and whole-body gestures. In this paper, we deliver a systematic literature review of this large body of work by summarizing the characteristics and findings ofN=216gesture elicitation studies subsuming 5,458 participants, 3,625 referents, and 148,340 elicited gestures. We highlight the descriptive, comparative, and generative virtues of our examination to provide practitioners with an effective method to (i) understand how new gesture elicitation studies position in the literature; (ii) compare studies from different authors; and (iii) identify opportunities for new research. We make our large corpus of papers accessible online as a Zotero group library at https://www.zotero.org/groups/2132650/gesture_elicitation_studies.

For over six decades, Fitts’s law (1954) has been utilized by researchers to quantify human pointing performance in terms of “throughput,” a combined speed-accuracy measure of aimed movement efficiency. Throughput measurements are commonly used to evaluate pointing techniques and devices, helping to inform software and hardware developments. Although Fitts’s law has been used extensively in HCI and beyond, its test-retest reliability, both in terms of throughput and model fit, from one session to the next, is still unexplored. Additionally, despite the fact that prior work has shown that Fitts’s law provides good model fits, with Pearson correlation coefficients commonly at r=.90 or above, the model fitness of Fitts’s law has not been thoroughly investigated for people who exhibit limited fine motor function in their dominant hand. To fill these gaps, we conducted a study with 21 participants with limited fine motor function and 34 participants without such limitations. Each participant performed a classic reciprocal pointing task comprising vertical ribbons in a 1-D layout in two sessions, which were at least four hours and at most 48 hours apart. Our findings indicate that the throughput values between the two sessions were statistically significantly different, both for people with and without limited fine motor function, suggesting that Fitts’s law provides low test-retest reliability. Importantly, the test-retest reliability of Fitts’s throughput metric was 4.7% lower for people with limited fine motor function. Additionally, we found that the model fitness of Fitts’s law as measured by Pearson correlation coefficient, r, was .89 (SD=0.08) for people without limited fine motor function, and .81 (SD=0.09) for people with limited fine motor function. Taken together, these results indicate that Fitts’s law should be used with caution and, if possible, over multiple sessions, especially when used in assistive technology evaluations.

Text editing on mobile devices can be a tedious process. To perform various editing operations, a user must repeatedly move his or her fingers between the text input area and the keyboard, making multiple round trips and breaking the flow of typing. In this work, we present Gedit, a system of on-keyboard gestures for convenient mobile text editing. Our design includes a ring gesture and flicks for cursor control, bezel gestures for mode switching, and four gesture shortcuts for copy, paste, cut, and undo. Variations of our gestures exist for one and two hands. We conducted an experiment to compare Gedit with the de facto touch+widget based editing interactions. Our results showed that Gedit's gestures were easy to learn, 24% and 17% faster than the de facto interactions for oneand two-handed use, respectively, and preferred by participants.

Correcting errors in entered text is a common task but usually diffcult to perform on mobile devices due to tedious cursor navigation steps. In this paper, we present JustCorrect, an intelligent post hoc text correction technique for smartphones. To make a correction, the user simply types the correct text at the end of their current input, and JustCorrect will automatically detect the error and apply the correction in the form of an insertion or a substitution. In this way, manual navigation steps are bypassed, and the correction can be committed with a single tap. We solved two critical problems to support JustCorrect: (1) Correction Algorithm: we propose an algorithm that infers the user's correction intention from the last typed word. (2) Input Modalities: our study revealed that both tap and gesture were suitable input modalities for performing JustCorrect. Based on our fndings, we integrated JustCorrect into a soft keyboard. Our user studies show that using JustCorrect reduces the text correction time by 12.8% over the stock Android keyboard and by 9.7% over the "Type, then Correct" text correction technique by Zhang et al. (2019). Overall, JustCorrect complements existing post hoc text correction techniques, making error correction more automatic and intelligent.

A digital calculator is one of the most frequently used touch screen applications. However, keypad-based character input in existing calculator applications requires precise, targeted key presses that are time-consuming and error-prone for many screen readers users. We introduce GestureCalc, a digital calculator that uses target-free gestures for arithmetic tasks. It allows eyes-free target-less input of digits and operations through taps and directional swipes with one to three fingers, guided by minimal audio feedback. We conducted a mixed methods longitudinal study with eight screen reader users and found that they entered characters with GestureCalc 40.5% faster on average than with a typical touch screen calculator. Participants made more mistakes but also corrected more errors with GestureCalc, resulting in 52.2% fewer erroneous calculations than the baseline. Over the three sessions in the study, participants were able to learn the GestureCalc gestures and efficiently perform short calculations. From our interviews after the second session, participants recognized the effort in learning a new gesture set, yet reported confidence in their ability to become fluent in practice.

A gender stereotype that has some basis in research is that men are more reluctant to ask for directions than women. We wanted to investigate whether this stereotype applies to technology-related contexts, affecting older adults' abilities to learn new technologies. To explore how help seeking and gender might relate for older adults, we conducted a controlled experiment with 36 individuals, of whom 18 identified as men and 18 identified as women, and observed how often they asked for help when learning new applications. We also conducted post-experiment interviews with participants. We found that although most participants stereotyped older men as being reluctant to ask for help in the interview, the gender difference was minimal in the experiment. Instead, individual differences had a greater effect: older participants took longer to complete tasks and participants with lower technology self-efficacy asked significantly more questions.

To understand whether people can identify their optimal touchscreen target sizes, we asked older and younger adults to identify optimal target sizes on a questionnaire and compared these chosen sizes to performance on a target acquisition task. We found that older individuals (60+) were better than younger adults at choosing their optimal target sizes. In fact, younger adults underestimated the smallest target size they could accurately touch by almost 6mm. This study suggests that older adults may be able to better configure target size settings than younger adults.

To investigate how older adults perceive ability changes (e.g., sensory, physical, cognitive) and how attitudes toward those changes affect perception and adoption of built-in mobile accessibility features (such as those found on Apple iOS and Google Android smartphones and tablets), we conducted an interview study with 14 older adults and six of their family members. Accessibility features were difficult for participants to find and configure, which were issues compounded by a reluctance to use trial-and-error. At 4-6 weeks after the interview, however, some participants had adopted new accessibility features that we had showed them, suggesting a willingness to adopt once features are made visible. The older adults who did already use accessibility features had experienced a disability earlier in life, suggesting that those experiencing progressive ability changes later in life might not be as aware of accessibility features, or might not have the know-how to adapt technologies to their changing needs. Our findings provide support for creating technologies that can detect older adults' abilities and recommend or enact interface changes to match.

Keypad-based character input in existing digital calculator applications on touch screen devices requires precise, targeted key presses that are time-consuming and error-prone for many screen reader users. We demonstrate GestureCalc, a digital calculator that uses target-free gestures for arithmetic tasks. It allows eyes-free target-less input of digits and operations through taps and directional swipes with one to three fingers, guided by minimal audio feedback. A study of the effectiveness of GestureCalc for screen reader users appears in a full paper by the authors at this conference.

End-user elicitation studies are a popular design method. Currently, such studies are usually confined to a lab, limiting the number and diversity of participants, and therefore the representativeness of their results. Furthermore, the quality of the results from such studies generally lacks any formal means of evaluation. In this paper, we address some of the limitations of elicitation studies through the creation of the Crowdlicit system along with the introduction of end-user identification studies, which are the reverse of elicitation studies. Crowdlicit is a new web-based system that enables researchers to conduct online and in-lab elicitation and identification studies. We used Crowdlicit to run a crowd-powered elicitation study based on Morris's "Web on the Wall" study (2012) with 78 participants, arriving at a set of symbols that included six new symbols different from Morris's. We evaluated the effectiveness of 49 symbols (43 from Morris and six from Crowdlicit) by conducting a crowd-powered identification study. We show that the Crowdlicit elicitation study resulted in a set of symbols that was significantly more identifiable than Morris's.

We present Cluster Touch, a combined user-independent and user-specific touch offset model that improves the accuracy of touch input on smartphones for people with motor impairments, and for people experiencing situational impairments while walking. Cluster Touch combines touch examples from multiple users to create a shared user-independent touch model, which is then updated with touch examples provided by an individual user to make it user-specific. Owing to this combination, Cluster Touch allows people to quickly improve the accuracy of their smartphones by providing only 20 touch examples. In a user study with 12 people with motor impairments and 12 people without motor impairments, but who were walking, Cluster Touch improved touch accuracy by 14.65% for the former group and 6.81% for the latter group over the native touch sensor. Furthermore, in an offline analysis of existing mobile interfaces, Cluster Touch improved touch accuracy by 8.21% and 4.84% over the native touch sensor for the two user groups, respectively.

Human-computer input performance inherently involves speed-accuracy tradeoffs---the faster users act, the more inaccurate those actions are. Therefore, comparing speeds and accuracies separately can result in ambiguous outcomes: Does a fast but inaccurate technique perform better or worse overall than a slow but accurate one? For pointing, speed and accuracy has been unified for over 60 years as throughput (bits/s) (Crossman 1957, Welford 1968), but to date, no similar metric has been established for text entry. In this paper, we introduce a text entry method-independent throughput metric based on Shannon information theory (1948). To explore the practical usability of the metric, we conducted an experiment in which 16 participants typed with a laptop keyboard using different cognitive sets, i.e., speed-accuracy biases. Our results show that as a performance metric, text entry throughput remains relatively stable under different speed-accuracy conditions. We also evaluated a smartphone keyboard with 12 participants, finding that throughput varied least compared to other text entry metrics. This work allows researchers to characterize text entry performance with a single unified measure of input efficiency.

Method-independent text entry evaluation tools are often used to conduct text entry experiments and compute performance metrics, like words per minute and error rates. The input stream paradigm of Soukoreff & MacKenzie (2001, 2003) still remains prevalent, which presents a string for transcription and uses a strictly serial character representation for encoding the text entry process. Although an advance over prior paradigms, the input stream paradigm is unable to support many modern text entry features. To address these limitations, we present transcription sequences: for each new input, a snapshot of the entire transcribed string unto that point is captured. By comparing adjacent strings within a transcription sequence, we can compute all prior metrics, reduce artificial constraints on text entry evaluations, and introduce new metrics. We conducted a study with 18 participants who typed 1620 phrases using a laptop keyboard, on-screen keyboard, and smartphone keyboard using features such as auto-correction, word prediction, and copy/paste. We also evaluated non-keyboard methods Dasher, gesture typing, and T9. Our results show that modern text entry methods and features can be accommodated, prior metrics can be correctly computed, and new metrics can reveal insights. We validated our algorithms using ground truth based on cursor positioning, confirming 100% accuracy. We also provide a new tool, TextTest++, to facilitate web-based evaluations.

Current text correction processes on mobile touch devices are laborious: users either extensively use backspace, or navigate the cursor to the error position, make a correction, and navigate back, usually by employing multiple taps or drags over small targets. In this paper, we present three novel text correction techniques to improve the correction process: Drag-n-Drop, Drag-n-Throw, and Magic Key. All of the techniques skip error-deletion and cursor-positioning procedures, and instead allow the user to type the correction first, and then apply that correction to a previously committed error. Specifically, Drag-n-Drop allows a user to drag a correction and drop it on the error position. Drag-n-Throw lets a user drag a correction from the keyboard suggestion list and "throw" it to the approximate area of the error text, with a neural network determining the most likely error in that area. Magic Key allows a user to type a correction and tap a designated key to highlight possible error candidates, which are also determined by a neural network. The user can navigate among these candidates by directionally dragging from atop the key, and can apply the correction by simply tapping the key. We evaluated these techniques in both text correction and text composition tasks. Our results show that correction with the new techniques was faster than de facto cursor and backspace-based correction. Our techniques apply to any touch-based text entry method.

We conduct the first large-scale analysis of the accessibility of mobile apps, examining what unique insights this can provide into the state of mobile app accessibility. We analyzed 5,753 free Android apps for label-based accessibility barriers in three classes of image-based buttons: Clickable Images, Image Buttons, and Floating Action Buttons. An epidemiology-inspired framework was used to structure the investigation. The population of free Android apps was assessed for label-based inaccessible button diseases. Three determinants of the disease were considered: missing labels, duplicate labels, and uninformative labels. The prevalence, or frequency of occurrences of barriers, was examined in apps and in classes of image-based buttons. In the app analysis, 35.9% of analyzed apps had 90% or more of their assessed image-based buttons labeled, 45.9% had less than 10% of assessed image-based buttons labeled, and the remaining apps were relatively uniformly distributed along the proportion of elements that were labeled. In the class analysis, 92.0% of Floating Action Buttons were found to have missing labels, compared to 54.7% of Image Buttons and 86.3% of Clickable Images. We discuss how these accessibility barriers are addressed in existing treatments, including accessibility development guidelines.

Personal technologies are rarely designed to be accessible to disabled people, partly due to the perceived challenge of including disability in design. Through design workshops, we addressed this challenge by infusing user-centered design activities with Design for Social Accessibility-a perspective emphasizing social aspects of accessibility-to investigate how professional designers can leverage social factors to include accessibility in design. We focused on how professional designers incorporated Design for Social Accessibility's three tenets: (1) to work with users with and without visual impairments; (2) to consider social and functional factors; (3) to employ tools-a framework and method cards-to raise awareness and prompt reflection on social aspects toward accessible design. We then interviewed designers about their workshop experiences. We found DSA to be an effective set of tools and strategies incorporating social/functional and non/disabled perspectives that helped designers create accessible design.

Breathalyzers, the standard quantitative method for assessing inebriation, are primarily owned by law enforcement and used only after a potentially inebriated individual is caught driving. However, not everyone has access to such specialized hardware. We present drunk user interfaces: smartphone user interfaces that measure how alcohol affects a person's motor coordination and cognition using performance metrics and sensor data. We examine five drunk user interfaces and combine them to form the "DUI app". DUI uses machine learning models trained on human performance metrics and sensor data to estimate a person's blood alcohol level (BAL). We evaluated DUI on 14 individuals in a week-long longitudinal study wherein each participant used DUI at various BALs. We found that with a global model that accounts for user-specific learning, DUI can estimate a person's BAL with an absolute mean error of 0.005% ± 0.007% and a Pearson's correlation coefficient of 0.96 with breathalyzer measurements.

Talk therapy is a common, effective, and desirable form of mental health treatment. Yet, it is inaccessible to many people. Enabling peers to chat online using effective principles of talk therapy could help scale this form of mental health care. To understand how such chats could be designed, we conducted a two-week field experiment with 40 people experiencing mental illnesses comparing two types of online chats-chats guided by prompts, and unguided chats. Results show that anxiety was significantly reduced from pre-test to post-test. User feedback revealed that guided chats provided solutions to problems and new perspectives, and were perceived as "deep," while unguided chats offered personal connection on shared experiences and were experienced as "smooth." We contribute the design of an online guided chat tool and insights into the design of peer support chat systems that guide users to initiate, maintain, and reciprocate emotional support.

Modern smartphones are built with capacitive-sensing touchscreens, which can detect anything that is conductive or has a dielectric differential with air. The human finger is an example of such a dielectric, and works wonderfully with such touchscreens. However, touch interactions are disrupted by raindrops, water smear, and wet fingers because capacitive touchscreens cannot distinguish finger touches from other conductive materials. When users' screens get wet, the screen's usability is significantly reduced. RainCheck addresses this hazard by filtering out potential touch points caused by water to differentiate fingertips from raindrops and water smear, adapting in real-time to restore successful interaction to the user. Specifically, RainCheck uses the low-level raw sensor data from touchscreen drivers and employs precise selection techniques to resolve water-fingertip ambiguity. Our study shows that RainCheck improves gesture accuracy by 75.7%, touch accuracy by 47.9%, and target selection time by 80.0%, making it a successful remedy to interference caused by rain and other water.

We introduce $Q, a super-quick, articulation-invariant point-cloud stroke-gesture recognizer for mobile, wearable, and embedded devices with low computing resources. $Q ran up to 142X faster than its predecessor $P in our benchmark evaluations on several mobile CPUs, and executed in less than 3% of $P's computations without any accuracy loss. In our most extreme evaluation demanding over 99% user-independent recognition accuracy, $P required 9.4s to run a single classification, while $Q completed in just 191ms (a 49X speed-up) on a Cortex-A7, one of the most widespread CPUs on the mobile market. $Q was even faster on a low-end 600-MHz processor, on which it executed in only 0.7% of $P's computations (a 142X speed-up), reducing classification time from two minutes to less than one second. $Q is the next major step for the "$-family" of gesture recognizers: articulation-invariant, extremely fast, accurate, and implementable on top of $P with just 30 extra lines of code.

End-user elicitation studies are a popular design method, but their data require substantial time and effort to analyze. In this paper, we present Crowdsensus, a crowd-powered tool that enables researchers to efficiently analyze the results of elicitation studies using subjective human judgment and automatic clustering algorithms. In addition to our own analysis, we asked six expert researchers with experience running and analyzing elicitation studies to analyze an end-user elicitation dataset of 10 functions for operating a web-browser, each with 43 voice commands elicited from end-users for a total of 430 voice commands. We used Crowdsensus to gather similarity judgments of these same 430 commands from 410 online crowd workers. The crowd outperformed the experts by arriving at the same results for seven of eight functions and resolving a function where the experts failed to agree. Also, using Crowdsensus was about four times faster than using experts.

Mobile accessibility is often a property considered at the level of a single mobile application (app), but rarely on a larger scale of the entire app "ecosystem," such as all apps in an app store, their companies, developers, and user influences. We present a novel conceptual framework for the accessibility of mobile apps inspired by epidemiology. It considers apps within their ecosystems, over time, and at a population level. Under this metaphor, "inaccessibility" is a set of diseases that can be viewed through an epidemiological lens. Accordingly, our framework puts forth notions like risk and protective factors, prevalence, and health indicators found within a population of apps. This new framing offers terminology, motivation, and techniques to reframe how we approach and measure app accessibility. It establishes how app accessibility can benefit from multi-factor, longitudinal, and population-based analyses. Our epidemiology-inspired conceptual framework is the main contribution of this work, intended to provoke thought and inspire new work enhancing app accessibility at a systemic level. In a preliminary exercising of our framework, we perform an analysis of the prevalence of common determinants or accessibility barriers. We assess the health of a stratified sample of 100 popular Android apps using Google's Accessibility Scanner. We find that 100% of apps have at least one of nine accessibility errors and examine which errors are most common. A preliminary analysis of the frequency of co-occurrences of multiple errors in a single app is also presented. We find 72% of apps have five or six errors, suggesting an interaction among different errors or an underlying influence.

We present Group Touch, a method for distinguishing among multiple users simultaneously interacting with a tabletop computer using only the touch information supplied by the device. Rather than tracking individual users for the duration of an activity, Group Touch distinguishes users from each other by modeling whether an interaction with the tabletop corresponds to either: (1) a new user, or (2) a change in users currently interacting with the tabletop. This reframing of the challenge as distinguishing users rather than tracking and identifying them allows Group Touch to support multi-user collaboration in real-world settings without custom instrumentation. Specifically, Group Touch examines pairs of touches and uses the difference in orientation, distance, and time between two touches to determine whether the same person performed both touches in the pair. Validated with field data from high-school students in a classroom setting, Group Touch distinguishes among users "in the wild" with a mean accuracy of 92.92% (SD=3.94%). Group Touch can imbue collaborative touch applications in real-world settings with the ability to distinguish among multiple users.

We present cascading dwell gaze typing, a novel approach to dwell-based eye typing that dynamically adjusts the dwell time of keys in an on-screen keyboard based on the likelihood that a key will be selected next, and the location of the key on the keyboard. Our approach makes unlikely keys more difficult to select and likely keys easier to select by increasing and decreasing their required dwell times, respectively. To maintain a smooth typing rhythm for the user, we cascade the dwell time of likely keys, slowly decreasing the minimum allowable dwell time as a user enters text. Cascading the dwell time affords users the benefits of faster dwell times while causing little disruption to users' typing cadence. Results from a longitudinal study with 17 non-disabled participants show that our dynamic cascading dwell technique was significantly faster than a static dwell approach. Participants were able to achieve typing speeds of 12.39 WPM on average with our cascading technique, whereas participants were able to achieve typing speeds of 10.62 WPM on average with a static dwell time approach. In a small evaluation conducted with five people with ALS, participants achieved average typing speeds of 9.51 WPM with our cascading dwell approach. These results show that our dynamic cascading dwell technique has the potential to improve gaze typing for users with and without disabilities.

We introduce interaction proxies as a strategy for runtime repair and enhancement of the accessibility of mobile applications. Conceptually, interaction proxies are inserted between an application's original interface and the manifest interface that a person uses to perceive and manipulate the application. This strategy allows third-party developers and researchers to modify an interaction without an application's source code, without rooting the phone, without otherwise modifying an application, while retaining all capabilities of the system (e.g., Android's full implementation of the TalkBack screen reader). This paper introduces interaction proxies, defines a design space of interaction re-mappings, identifies necessary implementation abstractions, presents details of implementing those abstractions in Android, and demonstrates a set of Android implementations of interaction proxies from throughout our design space. We then present a set of interviews with blind and low-vision people interacting with our prototype interaction proxies, using these interviews to explore the seamlessness of interaction, the perceived usefulness and potential of interaction proxies, and visions of how such enhancements could gain broad usage. By allowing third-party developers and researchers to improve an interaction, interaction proxies offer a new approach to personalizing mobile application accessibility and a new approach to catalyzing development, deployment, and evaluation of mobile accessibility enhancements.

Barriers to accessing mental health care leave the majority of people with mental illnesses without professional care. Peer support has been shown to address gaps in care, and could scale to wider audiences through technology. But technology design for mental health peer support lags far behind tools for individuals and clinicians. To identify opportunities for design, we interviewed 18 people with a diverse range of mental illnesses about their use of technology for peer support, and invited them to design technologies that could improve their experience of peer support. We found that technology could enhance peer support for mental health by: (1) matching peers on similarities beyond diagnosis; (2) enhancing accessibility; and (3) proactively mitigating risk through training and intervention. We discuss these findings in the context of the broad peer support literature, and present design opportunities for making mental health peer support tools empowering, accessible, and safe.

Research on children's interactions with touchscreen devices has examined small and large screens and compared interaction to adults or among children of different ages. Little work has explicitly compared interaction on different platforms, however. Large touchscreen displays can be deployed flat, as in a table, or vertically, as on a wall. While these two form factors have been studied, it is not known what differences may exist between them. We present a study of visitors to a science museum, including children and their parents, who interacted with Google Earth on either a touch table or a touch wall. We compare the types of gestures and interactions attempted on each device and find several interesting results, including: users of all ages tend to make standard touchscreen gestures on both platforms, but children were more likely than adults to try new gestures. Users were more likely to perform two-handed, multi-touch gestures on the touch wall than on the touch table. Our findings will inform the design of future interactive applications for each platform.

Despite practices addressing disability in design and advocating user-centered design (UCD) approaches, popular mainstream technologies remain largely inaccessible for people with disabilities. We conducted a design course study investigating how student designers regard disability and explored how designing for both disabled and non-disabled users encouraged students to think about accessibility throughout the design process. Students focused on a design project while learning UCD concepts and techniques, working with people with and without disabilities throughout the project. We found that designing for both disabled and non-disabled users surfaced challenges and tensions in finding solutions to satisfy both groups, influencing students' attitudes toward accessible design. In addressing these tensions, non-functional aspects of accessible design emerged as important complements to functional aspects for users with and without disabilities.

We present two contributions toward improving the accessibility of touch screens for people with motor impairments. First, we provide an exploration of the touch behaviors of 10 people with motor impairments, e.g., we describe how touching with the back or sides of the hand, with multiple fingers, or with knuckles creates varied multi-point touches. Second, we introduce Smart Touch, a novel template-matching technique for touch input that maps any number of arbitrary contact-areas to a user's intended (x,y) target location. The result is that users with motor impairments can touch however their abilities allow, and Smart Touch will resolve their intended touch point. Smart Touch therefore allows users to touch targets in whichever ways are most comfortable and natural for them. In an experimental evaluation, we found that Smart Touch predicted (x,y) coordinates of the users' intended target locations over three times closer to the intended target than the native Land-on and Lift-off techniques reported by the built-in touch sensors found in the Microsoft PixelSense interactive tabletop. This result is an important step toward improving touch accuracy for people with motor impairments and others for whom touch screen operation was previously impossible.

Elicitation studies, where users supply proposals meant to effect system commands, have become a popular method for system designers. But the method to date has assumed a within-subjects procedure and statistics. Despite the benefits of examining the relative agreement of independent groups (e.g., men versus women, children versus adults, novices versus experts, etc.), the lack of appropriate tools for between-subjects agreement rate analysis have prevented so far such comparative investigations. In this work, we expand the elicitation method to between-subjects designs. We introduce a new measure for evaluating coagreement between groups and a new statistical test for agreement rate analysis that reports the exact p-value to evaluate the significance of the difference between agreement rates calculated for independent groups. We show the usefulness of our tools by re-examining previously published gesture elicitation data, for which we discuss significant differences in agreement for technical and non-technical participants, men and women, and different acquisition technologies. Our new tools will enable practitioners to properly analyze their user-elicited data resulted from complex experimental designs with multiple independent groups and, consequently, will help them understand agreement data and verify hypotheses about agreement at more sophisticated levels of analysis.

Interaction logs generated by educational software can provide valuable insights into the collaborative learning process and identify opportunities for technology to provide adaptive assistance. Modeling collaborative learning processes at tabletop computers is challenging, as the computer is only able to log a portion of the collaboration, namely the touch events on the table. Our previous lab study with adults showed that patterns in a group's touch interactions with a tabletop computer can reveal the quality of aspects of their collaborative process. We extend this understanding of the relationship between touch interactions and the collaborative process to adolescent learners in a field setting and demonstrate that the touch patterns reflect the quality of collaboration more broadly than previously thought, with accuracies up to 84.2%. We also present an approach to using the touch patterns to model the quality of collaboration in real-time.

As we increasingly strive for scientific rigor and generalizability in HCI research, should we entertain any hope that by doing good science, our discoveries will eventually be more transferrable to industry? We present an in-depth case study of how an HCI research innovation goes through the process of transitioning from a university project to a revenue-generating startup financed by venture capital. The innovation is a novel contextual help system for the Web, and we reflect on the different methods used to evaluate it and how research insights endure attempted dissemination as a commercial product. Although the extent to which any innovation succeeds commercially depends on a number of factors like market forces, we found that our HCI innovation with user-centered origins was in a unique position to gain traction with customers and garner buy-in from investors. However, since end users were not the buyers of our product, a strong user-centered focus obfuscated other critical needs of the startup and pushed out perspectives of non-user-centered stakeholders. To make the research-to-product transition, we had to focus on adoption-centered design, the process of understanding and designing for adopters and stakeholders of the product. Our case study raises questions about how we evaluate the novelty and research contributions of HCI innovations with respect to their potential for commercial impact.

Smartphones and tablets are often used in dynamic environments that force users to break focus and attend to their surroundings, creating a form of "situational impairment." Current mobile devices have no ability to sense when users divert or restore their attention, let alone provide support for resuming tasks. We therefore introduce SwitchBack, a system that allows mobile device users to resume tasks more efficiently. SwitchBack is built upon Focus and Saccade Tracking (FAST), which uses the front-facing camera to determine when the user is looking and how their eyes are moving across the screen. In a controlled study, we found that FAST can identify how many lines the user has read in a body of text within a mean absolute percent error of just 3.9%. We then tested SwitchBack in a dual focus-of-attention task, finding that SwitchBack improved average reading speed by 7.7% in the presence of distractions.

We address in this work the process of agreement rate analysis for characterizing the level of consensus between participants' proposals elicited during guessability studies. Two new measures, i.e., disagreement rate for referents and coagreement rate between referents, are proposed to accompany the widely-used agreement rate formula of Wobbrock et al. [37] when reporting participants' consensus for symbolic input. A statistical significance test for comparing the agreement rates of k>=2 referents is presented in analogy with Cochran's success/failure Q test [5], for which we express the test statistic in terms of agreement and coagreement rates. We deliver a toolkit to assist practitioners to compute agreement, disagreement, and coagreement rates, and run statistical tests for agreement rates at p=.05, .01, and .001 levels of significance. We validate our theoretical development of agreement rate analysis in relation with several previously published elicitation studies. For example, when we present the probability distribution function of the agreement rate measure, we also use it (1) to explain the magnitude of agreement rates previously reported in the literature, and (2) to propose qualitative interpretations for agreement rates, in analogy with Cohen's guidelines for effect sizes [6]. We also re-examine previously published elicitation data from the perspective of the agreement rate test statistic, and highlight new findings on the effect of referents over agreement rates, unattainable prior to this work. We hope that our contributions will advance the current knowledge in agreement rate analysis, providing researchers and practitioners with new techniques and tools to help them understand user-elicited data at deeper levels of detail and sophistication.

Mobile sign language video communication has the potential to be more accessible and affordable if the current recommended video transmission standard of 25 frames per second at 100 kilobits per second (kbps) as prescribed in the International Telecommunication Standardization Sector (ITU-T) Q.26/16 were relaxed. To investigate sign language video intelligibility at lower settings, we conducted a laboratory study, where fluent ASL signers in pairs held real-time free-form conversations over an experimental smartphone app transmitting real-time video at 5 fps/25 kbps, 10 fps/50 kbps, 15 fps/75 kbps, and 30 fps/150 kbps, settings well below the ITU-T standard that save both bandwidth and battery life. The aim of the laboratory study was to investigate how fluent ASL signers adapt to the lower video transmission rates, and to identify a lower threshold at which intelligible real-time conversations could be held. We gathered both subjective and objective measures from participants and calculated battery life drain. As expected, reducing the frame rate/bit rate monotonically extended the battery life. We discovered all participants were successful in holding intelligible conversations across all frame rates/bit rates. Participants did perceive the lower quality of video transmitted at 5 fps/25 kbps and felt that they were signing more slowly to compensate; however, participants' rate of fingerspelling did not actually decrease. This and other findings support our recommendation that intelligible mobile sign language conversations can occur at frame rates as low as 10 fps/50 kbps while optimizing resource consumption, video intelligibility, and user preferences.

We present the Bubble Lens, a new target acquisition technique that remedies the limitations of the Bubble Cursor to increase the speed and accuracy of acquiring small, dense targets--precisely those targets for which the Bubble Cursor degenerates to a point cursor. When targets are large and sparse, the Bubble Lens behaves like the Bubble Cursor. But when targets are small and dense, the Bubble Lens automatically magnifies nearby targets, making them larger in both visual- and motor-space. Importantly, magnification is not governed by an explicit user-invoked mode-switch. Rather, magnification is activated through kinematic triggering, a technique that continuously examines an unfolding velocity profile to automatically trigger mode changes based on observed features. In a first study, we found the Bubble Cursor performed poorly when targets had an effective size smaller than 10 pixels. Using this threshold for the Bubble Lens in a second study, we found that the Bubble Lens significantly outperformed the Bubble Cursor, decreasing movement time by 10.2% and error rates by 37.9%, making the Bubble Lens the fastest current pointing technique.

We present a new method of predicting the endpoints of mouse movements. While prior approaches to endpoint prediction have relied upon normative kinematic laws, regression, or control theory, our approach is straightforward but kinematically rich. Our key insight is to regard the unfolding velocity profile of a pointing movement as a 2-D stroke gesture and to use template matching to predict the endpoint based on prior observed movements. We call our technique kinematic template matching (KTM), which is simple to implement, user-adaptable, and kinematically expressive. In a study of 17 able-bodied participants evaluated over movement amplitudes ranging from 100-800 pixels, we found KTM to predict endpoints that were within 83 pixels of the true endpoint at 50% of the way through the movement, within 48 pixels at 75%, and within 39 pixels at 90%, using 1000 templates per participant. These accuracies make KTM as successful an approach to endpoint prediction as any prior technique, while being easier to implement and understand than most.

We introduce gesture heatmaps, a novel gesture analysis technique that employs color maps to visualize the variation of local features along the gesture path. Beyond current gesture analysis practices that characterize gesture articulations with single-value descriptors, e.g., size, path length, or speed, gesture heatmaps are able to show with colorful visualizations how the value of any such descriptors vary along the gesture path. We evaluate gesture heatmaps on three public datasets comprising 15,840 gesture samples of 70 gesture types from 45 participants, on which we demonstrate heatmaps' capabilities to (1) explain causes for recognition errors, (2) characterize users' gesture articulation patterns under various conditions, e.g., finger versus pen gestures, and (3) help understand users' subjective perceptions of gesture commands, such as why some gestures are perceived easier to execute than others. We also introduce chromatic confusion matrices that employ gesture heatmaps to extend the expressiveness of standard confusion matrices to better understand gesture classification performance. We believe that gesture heatmaps will prove useful to researchers and practitioners doing gesture analysis, and consequently, they will inform the design of better gesture sets and development of more accurate recognizers.

Adding tactile feedback to touch screens can improve their accessibility to blind users, but prior approaches to integrating tactile feedback with touch screens have either offered limited functionality or required extensive (and typically expensive) customization of the hardware. We introduce touchplates, carefully designed tactile guides that provide tactile feedback for touch screens in the form of physical guides that are overlaid on the screen and recognized by the underlying application. Unlike prior approaches to integrating tactile feedback with touch screens, touchplates are implemented with simple plastics and use standard touch screen software, making them versatile and inexpensive. Touchplates may be customized to suit individual users and applications, and may be produced on a laser cutter, 3D printer, or made by hand. We describe the design and implementation of touchplates, a "starter kit" of touchplates, and feedback from a formative evaluation with 9 people with visual impairments. Touchplates provide a low-cost, adaptable, and accessible method of adding tactile feedback to touch screen interfaces.

Mobile sign language video conversations can become unintelligible due to high video transmission rates causing network congestion and delayed video. In an effort to understand how much sign language video quality can be sacrificed, we evaluated the perceived lower limits of intelligible sign language video transmitted at four low frame rates (1, 5, 10, and 15 frames per second [fps]) and four low fixed bitrates (15, 30, 60, and 120 kilobits per second [kbps]). We discovered an "intelligibility ceiling effect," where increasing the frame rate above 10 fps decreased perceived intelligibility, and increasing the bitrate above 60 kbps produced diminishing returns. Additional findings suggest that relaxing the recommended international video transmission rate, 25 fps at 100 kbps or higher, would still provide intelligible content while considering network resources and bandwidth consumption. As part of this work, we developed the Human Signal Intelligibility Model, a new conceptual model useful for informing evaluations of video intelligibility.

We present a multi-site field study to evaluate LemonAid, a crowdsourced contextual help approach that allows users to retrieve relevant questions and answers by making selections within the interface. We deployed LemonAid on 4 different web sites used by thousands of users and collected data over several weeks, gathering over 1,200 usage logs, 168 exit surveys, and 36 one-on-one interviews. Our results indicate that over 70% of users found LemonAid to be helpful, intuitive, and desirable for reuse. Software teams found LemonAid easy to integrate with their sites and found the analytics data aggregated by LemonAid a novel way of learning about users' popular questions. Our work provides the first holistic picture of the adoption and use of a crowdsourced contextual help system and offers several insights into the social and organizational dimensions of implementing such help systems for real-world applications.

Despite the apparent popularity of touchscreens for older adults, little is known about the psychomotor performance of these devices. We compared performance between older adults and younger adults on four desktop and touchscreen tasks: pointing, dragging, crossing and steering. On the touchscreen, we also examined pinch-to-zoom. Our results show that while older adults were significantly slower than younger adults in general, the touchscreen reduced this performance gap relative to the desktop and mouse. Indeed, the touchscreen resulted in a significant movement time reduction of 35% over the mouse for older adults, compared to only 16% for younger adults. Error rates also decreased.

The challenge of mobile text entry is exacerbated as mobile devices are used in a number of situations and with a number of hand postures. We introduce ContextType, an adaptive text entry system that leverages information about a user's hand posture (using two thumbs, the left thumb, the right thumb, or the index finger) to improve mobile touch screen text entry. ContextType switches between various keyboard models based on hand posture inference while typing. ContextType combines the user's posture-specific touch pattern information with a language model to classify the user's touch events as pressed keys. To create our models, we collected usage patterns from 16 participants in each of the four postures. In a subsequent study with the same 16 participants comparing ContextType to a control condition, ContextType reduced total text entry error rate by 20.6%.

Gesture-based touch screen user interfaces, when designed to be accessible to blind users, can be an effective mode of interaction for those users. However, current accessible touch screen interaction techniques suffer from one serious limitation: they are only usable on devices that have been explicitly designed to support them. Access Lens is a new interaction method that uses computer vision-based gesture tracking to enable blind people to use accessible gestures on paper documents and other physical objects, such as product packages, device screens, and home appliances. This paper describes the development of Access Lens hardware and software, the iterative design of Access Lens in collaboration with blind computer users, and opportunities for future development.

A "discount" version of Q-methodology for HCI, called "HCI-Q", can be used in iterative design cycles to explore, from the point of view of users and other stakeholders, what makes technologies personally significant. Initially, designers critically reflect on their own assumptions about how a design may affect social and individual behavior. Then, designers use these assumptions as stimuli to elicit other people's points of view. This process of critical self-reflection and evaluation helps the designer to assess the fit between a design and its intended social context of use. To demonstrate the utility of HCI-Q for research and design, we use HCI-Q to explore stakeholders' responses to a prototype Alternative and Augmentative Communication (AAC) application called Vid2Speech. We show that our adaptation of Q-methodology is useful for revealing the structure of consensus and conflict among stakeholder perspectives, helping to situate design within the context of relevant value tensions and norms.

Current measures of stroke gesture articulation lack descriptive power because they only capture absolute characteristics about the gesture as a whole, not fine-grained features that reveal subtleties about the gesture articulation path. We present a set of twelve new relative accuracy measures for stroke gesture articulation that characterize the geometric, kinematic, and articulation accuracy of single and multi-stroke gestures. To compute the accuracy measures, we introduce the concept of a gesture task axis. We evaluate our measures on five public datasets comprising 38,245 samples from 107 participants, about which we make new discoveries; e.g., gestures articulated at fast speed are shorter in path length than slow or medium-speed gestures, but their path lengths vary the most, a finding that helps understand recognition performance. This work will enable a better understanding of users' stroke gesture articulation behavior, ultimately leading to better gesture set designs and more accurate recognizers.

Blind mobile device users face security risks such as inaccessible authentication methods, and aural and visual eavesdropping. We interviewed 13 blind smartphone users and found that most participants were unaware of or not concerned about potential security threats. Not a single participant used optional authentication methods such as a password-protected screen lock. We addressed the high risk of unauthorized user access by developing PassChords, a non-visual authentication method for touch surfaces that is robust to aural and visual eavesdropping. A user enters a PassChord by tapping several times on a touch surface with one or more fingers. The set of fingers used in each tap defines the password. We give preliminary evidence that a four-tap PassChord has about the same entropy, a measure of password strength, as a four-digit personal identification number (PIN) used in the iPhone's Passcode Lock. We conducted a study with 16 blind participants that showed that PassChords were nearly three times as fast as iPhone's Passcode Lock with VoiceOver, suggesting that PassChords are a viable accessible authentication method for touch screens.

Enabling end-users of Augmentative and Alternative Communication (AAC) systems to add personalized video content at runtime holds promise for improving communication, but the requirements for such systems are as yet unclear. To explore this issue, we present Vid2Speech, a prototype AAC system for children with complex communication needs (CCN) that uses personalized video to enhance representations of action words. We describe three design goals that guided the integration of personalized video to enhance AAC in our early-stage prototype: 1) Providing social-temporal navigation; 2) Enhancing comprehension; and 3) Enabling customization in real time. Our system concept represents one approach to realizing these goals, however, we contribute the goals and the system as a starting point for future innovations in personalized video-based AAC.

Web-based technical support such as discussion forums and social networking sites have been successful at ensuring that most technical support questions eventually receive helpful answers. Unfortunately, finding these answers is still quite difficult, since users' textual queries are often incomplete, imprecise, or use different vocabularies to describe the same problem. We present LemonAid, a new approach to help that allows users to find help by instead selecting a label, widget, link, image or other user interface (UI) element that they believe is relevant to their problem. LemonAid uses this selection to retrieve previously asked questions and their corresponding answers. The key insight that makes LemonAid work is that users tend to make similar selections in the interface for similar help needs and different selections for different help needs. Our initial evaluation shows that across a corpus of dozens of tasks and thousands of requests, LemonAid retrieved a result for 90% of help requests based on UI selections and, of those, over half had relevant matches in the top 2 results.

We present a general-purpose implementation of a target aware pointing technique, functional across an entire desktop and independent of application implementations. Specifically, we implement Grossman and Balakrishnan's Bubble Cursor, the fastest general pointing facilitation technique in the literature. Our implementation obtains the necessary knowledge of interface targets using a combination of pixel-level analysis and social annotation. We discuss the most novel aspects of our implementation, including methods for interactive creation and correction of pixel-level prototypes of interface elements and methods for interactive annotation of how the cursor should select identified elements. We also report on limitations of the Bubble Cursor unearthed by examining our implementation in the complexity of real-world interfaces. We therefore contribute important progress toward real-world deployment of an important family of techniques and shed light on the gap between understanding techniques in controlled settings versus behavior with real-world interfaces.

We present the Input Observer, a tool that can run quietly in the background of users' computers and measure their text entry and mouse pointing performance from everyday use. In lab studies, participants are presented with prescribed tasks, enabling easy identification of speeds and errors. In everyday use, no such prescriptions exist. We devised novel algorithms to segment text entry and mouse pointing input streams into "trials". We are the first to measure errors for unprescribed text entry and mouse pointing. To measure errors, we utilize web search engines, adaptive offline dictionaries, an Automation API, and crowdsourcing. Capturing errors allows us to employ Crossman's (1957) speed-accuracy normalization when calculating Fitts' law throughputs. To validate the Input Observer, we compared its measures from 12 participants over a week of computer use to the same participants' results from a lab study. Overall, in the lab and field, average text entry speeds were 74.47 WPM and 80.59 WPM, respectively. Average uncorrected error rates were near zero, at 0.12% and 0.28%. For mouse pointing, average movement times were 971 ms and 870 ms. Average pointing error rates were 4.42% and 4.66%. Average throughputs were 3.48 bits/s and 3.45 bits/s. Device makers, researchers, and assistive technology specialists may benefit from measures of everyday use.

Although many techniques have been proposed to improve text input on touch screens, the vast majority of this research ignores non-alphanumeric input (i.e., punctuation, symbols, and modifiers). To support this input, widely adopted commercial touch-screen interfaces require mode switches to alternate keyboard layouts for most punctuation and symbols. Our approach is to augment existing ten-finger QWERTY keyboards with multi-touch gestural input that can exist as a complement to the moded-keyboard approach. To inform our design, we conducted a study to elicit user-defined gestures from 20 participants. The final gesture set includes both multi-touch and single-touch gestures for commonly used non-alphanumeric text input. We implemented and conducted a preliminary evaluation of a touch-screen keyboard augmented with this technique. Findings show that using gestures for non-alphanumeric input is no slower than using keys, and that users strongly prefer gestures to a moded-keyboard interface.

Although typing on touchscreens is slower than typing on physical keyboards, touchscreens offer a critical potential advantage: they are software-based, and, as such, the keyboard layout and classification models used to interpret key presses can dynamically adapt to suit each user's typing pattern. To explore this potential, we introduce and evaluate two novel personalized keyboard interfaces, both of which adapt their underlying key-press classification models. The first keyboard also visually adapts the location of keys while the second one always maintains a visually stable rectangular layout. A three-session user evaluation showed that the keyboard with the stable rectangular layout significantly improved typing speed compared to a control condition with no personalization. Although no similar benefit was found for the keyboard that also offered visual adaptation, overall subjective response to both new touchscreen keyboards was positive. As personalized keyboards are still an emerging area of research, we also outline a design space that includes dimensions of adaptation and key-press classification features.

The lack of tactile feedback on touch screens makes typing difficult, a challenge exacerbated when situational impairments like walking vibration and divided attention arise in mobile settings. We introduce WalkType, an adaptive text entry system that leverages the mobile device's built-in tri-axis accelerometer to compensate for extraneous movement while walking. WalkType's classification model uses the displacement and acceleration of the device, and inference about the user's footsteps. Additionally, WalkType models finger-touch location and finger distance traveled on the screen, features that increase overall accuracy regardless of movement. The final model was built on typing data collected from 16 participants. In a study comparing WalkType to a control condition, WalkType reduced uncorrected errors by 45.2% and increased typing speed by 12.9% for walking participants.

Rapid prototyping of gesture interaction for emerging touch platforms requires that developers have access to fast, simple, and accurate gesture recognition approaches. The $-family of recognizers ($1, $N) addresses this need, but the current most advanced of these, $N-Protractor, has significant memory and execution costs due to its combinatoric gesture representation approach. We present $P, a new member of the $-family, that remedies this limitation by considering gestures as clouds of points. $P performs similarly to $1 on unistrokes and is superior to $N on multistrokes. Specifically, $P delivers >99% accuracy in user-dependent testing with 5+ training samples per gesture type and stays above 99% for user-independent tests when using data from 10 participants. We provide a pseudocode listing of $P to assist developers in porting it to their specific platform and a "cheat sheet" to aid developers in selecting the best member of the $-family for their specific application needs.

We introduce GripSense, a system that leverages mobile device touchscreens and their built-in inertial sensors and vibration motor to infer hand postures including one- or two-handed interaction, use of thumb or index finger, or use on a table. GripSense also senses the amount of pres-sure a user exerts on the touchscreen despite a lack of direct pressure sensors by inferring from gyroscope readings when the vibration motor is "pulsed." In a controlled study with 10 participants, GripSense accurately differentiated device usage on a table vs. in hand with 99.67% accuracy and when in hand, it inferred hand postures with 84.26% accuracy. In addition, GripSense distinguished three levels of pressure with 95.1% accuracy. A usability analysis of GripSense was conducted in three custom applications and showed that pressure input and hand-posture sensing can be useful in a number of scenarios.

We explore using vibration on a smartphone to provide turn-by-turn walking instructions to people with visual impairments. We present two novel feedback methods called Wand and ScreenEdge and compare them to a third method called Pattern. We built a prototype and conducted a user study where 8 participants walked along a pre-programmed route using the 3 vibration feedback methods and no audio output. Participants interpreted the feedback with an average error rate of just 4 percent. Most preferred the Pattern method, where patterns of vibrations indicate different directions, or the ScreenEdge method, where areas of the screen correspond to directions and touching them may induce vibration.

Video and image quality are often objectively measured using peak signal-to-noise ratio (PSNR), but for sign language video, human comprehension is most important. Yet the relationship of human comprehension to PSNR has not been studied. In this survey, we determine how well PSNR matches human comprehension of sign language video. We use very low bitrates (10-60 kbps) and two low spatial resolutions (192×144 and 320×240 pixels) which may be typical of video transmission on mobile phones using 3G networks. In a national online video-based user survey of 103 respondents, we found that respondents preferred the 320×240 spatial resolution transmitted at 20 kbps and higher; this does not match what PSNR results would predict. However, when comparing perceived ease/difficulty of comprehension, we found that responses did correlate well with measured PSNR. This suggests that PSNR may not be suitable for representing subjective video quality, but can be reliable as a measure for comprehensibility of American Sign Language (ASL) video. These findings are applied to our experimental mobile phone application, MobileASL, which enables real-time sign language communication for Deaf users at low bandwidths over the U.S. 3G cellular network.

Blind and deaf-blind people often rely on public transit for everyday mobility, but using transit can be challenging for them. We conducted semi-structured interviews with 13 blind and deaf-blind people to understand how they use public transit and what human values were important to them in this domain. Two key values were identified: independence and safety. We developed GoBraille, two related Braille-based applications that provide information about buses and bus stops while supporting the key values. GoBraille is built on MoBraille, a novel framework that enables a Braille display to benefit from many features in a smartphone without knowledge of proprietary, device-specific protocols. Finally, we conducted user studies with blind people to demonstrate that GoBraille enables people to travel more independently and safely. We also conducted co-design with a deaf-blind person, finding that a minimalist interface, with short input and output messages, was most effective for this population.

Despite the growing research on usability in the pre-development phase, we know little about post-deployment usability activities. To characterize these activities, we surveyed 333 full-time usability professionals and consultants working in large and small corporations from a wide range of industries. Our results show that, as a whole, usability professionals are currently not playing a substantial role in the post-deployment phase compared to other phases of user-centered design, but when they do, practitioners find their interactions quite valuable. We highlight opportunities in HCI research and practice to bridge this gap by working more closely with software support and maintenance teams. We also raise the need to understand what might be called 'usability maintenance,' that is, the process and procedures, by which usability is maintained after deployment.

Touch screen surfaces large enough for ten-finger input have become increasingly popular, yet typing on touch screens pales in comparison to physical keyboards. We examine typing patterns that emerge when expert users of physical keyboards touch-type on a flat surface. Our aim is to inform future designs of touch screen keyboards, with the ultimate goal of supporting touch-typing with limited tactile feedback. To study the issues inherent to flat-glass typing, we asked 20 expert typists to enter text under three conditions: (1) with no visual keyboard and no feedback on input errors, then (2) with and (3) without a visual keyboard, but with some feedback. We analyzed touch contact points and hand contours, looking at attributes such as natural finger positioning, the spread of hits among individual keys, and the pattern of non-finger touches. We also show that expert typists exhibit spatially consistent key press distributions within an individual, which provides evidence that eyes-free touch-typing may be possible on touch surfaces and points to the role of personalization in such a solution. We conclude with implications for design.

Despite growing awareness of the accessibility issues surrounding touch screen use by blind people, designers still face challenges when creating accessible touch screen interfaces. One major stumbling block is a lack of understanding about how blind people actually use touch screens. We conducted two user studies that compared how blind people and sighted people use touch screen gestures. First, we conducted a gesture elicitation study in which 10 blind and 10 sighted people invented gestures to perform common computing tasks on a tablet PC. We found that blind people have different gesture preferences than sighted people, including preferences for edge-based gestures and gestures that involve tapping virtual keys on a keyboard. Second, we conducted a performance study in which the same participants performed a set of reference gestures. We found significant differences in the speed, size, and shape of gestures performed by blind people versus those performed by sighted people. Our results suggest new design guidelines for accessible touch screen interfaces.

Few research studies focus on how the use of assistive technologies is affected by social interaction among people. We present an interview study of 20 individuals to determine how assistive technology use is affected by social and professional contexts and interactions. We found that specific assistive devices sometimes marked their users as having disabilities; that functional access took priority over feeling self-conscious when using assistive technologies; and that two misperceptions pervaded assistive technology use: (1) that assistive devices could functionally eliminate a disability, and (2) that people with disabilities would be helpless without their devices. Our findings provide further evidence that accessibility should be built into mainstream technologies. When this is not feasible, assistive devices should incorporate cutting edge technologies and strive to be designed for social acceptability, a new design approach we propose here.

Nonparametric data from multi-factor experiments arise often in human-computer interaction (HCI). Examples may include error counts, Likert responses, and preference tallies. But because multiple factors are involved, common nonparametric tests (e.g., Friedman) are inadequate, as they are unable to examine interaction effects. While some statistical techniques exist to handle such data, these techniques are not widely available and are complex. To address these concerns, we present the Aligned Rank Transform (ART) for nonparametric factorial data analysis in HCI. The ART relies on a preprocessing step that "aligns" data before applying averaged ranks, after which point common ANOVA procedures can be used, making the ART accessible to anyone familiar with the F-test. Unlike most articles on the ART, which only address two factors, we generalize the ART to N factors. We also provide ARTool and ARTweb, desktop and Web-based programs for aligning and ranking data. Our re-examination of some published HCI results exhibits advantages of the ART.

Recently, Wobbrock et al. (2008) derived a predictive model of pointing accuracy to complement Fitts' law's predictive model of pointing speed. However, their model was based on one-dimensional (1-D) horizontal movement, while applications of such a model require two dimensions (2-D). In this paper, the pointing error model is investigated for 2-D pointing in a study of 21 participants performing a time-matching task on the ISO 9241-9 ring-of-circles layout. Results show that the pointing error model holds well in 2-D. If univariate endpoint deviation (SDx) is used, regressing on N=72 observed vs. predicted error rate points yields R2=.953. If bivariate endpoint deviation (SDx,y) is used, regression yields R2=.936. For both univariate and bivariate models, the magnitudes of observed and predicted error rates are comparable.

Fitts' law (1954) characterizes pointing speed-accuracy performance as throughput, whose invariance to target distances (A) and sizes (W) is known. However, it is unknown whether throughput and Fitts' law models in general are invariant to task dimensionality (1-D vs. 2-D), whether univariate (SDx) or bivariate (SDx,y) endpoint deviation is used, whether throughput is calculated using the mean-of-means approach or the slope-inverse approach, or whether Guiard's (2009) Form - Scale experiment design is used instead of fully crossed A-W factors. We empirically investigate the confluence of these issues, finding that Fitts' law is largely invariant across 1-D and 2-D, provided that univariate endpoint deviation (SDx) is used in both, but that for 2-D pointing data, bivariate endpoint deviation (SDx,y) results in better Fitts' law models. Also, the mean-of-means throughput calculation exhibits lower variance across subjects and dimensionalities than the slope-inverse calculation. In light of these and other findings, we offer recommendations for pointing evaluations, especially in 2-D. We also offer an evaluation tool called Fitts Study to facilitate comparisons.

We present Portico, a portable system for enabling tangible interaction on and around tablet computers. Two cameras on small foldable arms are positioned above the display to recognize a variety of physical objects placed on or around the tablet. These cameras have a larger field-of-view than the screen, allowing Portico to extend interaction significantly beyond the tablet itself. Our prototype, which uses a 12" tablet, delivers an interaction space six times the size of the tablet screen. Portico thus allows tablets to extend both their sensing capabilities and interaction space without sacrificing portability. We describe the design of our system and present a number of applications that demonstrate Portico's unique capability to track objects. We focus on a number of fun applications that demonstrate how such a device can be used as a low-cost way to create personal surface computing experiences. Finally, we discuss the challenges in supporting tangible interaction beyond the screen and describe possible mechanisms for overcoming them.

Many touch screens remain inaccessible to blind users, and those approaches to providing access that do exist offer minimal support for interacting with large touch screens or spatial data. In this paper, we introduce a set of three software-based access overlays intended to improve the accessibility of large touch screen interfaces, specifically interactive tabletops. Our access overlays are called edge projection, neighborhood browsing, and touch-and-speak. In a user study, 14 blind users compared access overlays to an implementation of Apple's VoiceOver screen reader. Our results show that two of our techniques were faster than VoiceOver, that participants correctly answered more questions about the screen's layout using our techniques, and that participants overwhelmingly preferred our techniques. We developed several applications demonstrating the use of access overlays, including an accessible map kiosk and an accessible board game.

MobileASL is a video compression project for two-way, real-time video communication on cell phones, allowing Deaf people to communicate in the language most accessible to them, American Sign Language. Unfortunately, running MobileASL quickly depletes a full battery charge in a few hours. Previous work on MobileASL investigated a method called variable frame rate (VFR) to increase the battery duration. We expand on this previous work by creating two new power saving algorithms, variable spatial resolution (VSR), and the application of both VFR and VSR. These algorithms extend the battery life by altering the temporal and/or spatial resolutions of video transmitted on MobileASL. We found that implementing only VFR extended the battery life from 284 minutes to 307 minutes; implementing only VSR extended the battery life to 306 minutes, and implementing both VFR and VSR extended the battery life to 315 minutes. We evaluated all three algorithms by creating a linguistically accessible online survey to investigate Deaf people's perceptions of video quality when these algorithms were applied. In our survey results, we found that VFR produces perceived video choppiness and VSR produces perceived video blurriness; however, a surprising finding was that when both VFR and VSR are used together, they largely ameliorate the choppiness and blurriness perceived, i.e., they each improve the use of the other. This is a useful finding because using VFR and VSR together saves the most battery life.

Although usability methods are widely used for evaluating conventional graphical user interfaces and websites, there is a growing concern that current approaches are inadequate for evaluating complex, domain-specific tools. We interviewed 21 experienced usability professionals, including in-house experts, external consultants, and managers working in a variety of complex domains, and uncovered the challenges commonly posed by domain complexity and how practitioners work around them. We found that despite the best efforts by usability professionals to get familiar with complex domains on their own, the lack of formal domain expertise can be a significant hurdle for carrying out effective usability evaluations. Partnerships with domain experts lead to effective results as long as domain experts are willing to be an integral part of the usability team. These findings suggest that for achieving usability in complex domains, some fundamental educational changes may be needed in the training of usability professionals.

We present a study that evaluates the effectiveness of augmenting on-screen instructions with micro-projection for manual task guidance unlike prior work, which replaced screen instructions with alternative modalities (e.g., head-mounted displays). In our study, 30 participants completed 10 trials each of 11 manual tasks chosen to represent a set of common task-components (e.g., cutting, folding) found in many everyday activities such as crafts, cooking, and hobby electronics. Fifteen participants received only on-screen instructions, and 15 received both on-screen and micro-projected instructions. In contrast to prior work, which focused only on whole tasks, our study examines the benefit of augmenting common task instructions. The augmented instructions improved participants' performance overall; however, we show that in certain cases when projected guides and physical objects visually interfered, projected elements caused increased errors. Our results demonstrate that examining effectiveness at an instruction level is both useful and necessary, and provide insight into the design of systems that help users perform everyday tasks.

Computer users with motor impairments face major challenges with conventional mouse pointing. These challenges are mostly due to fine pointing corrections at the final stages of target acquisition. To reduce the need for correction-phase pointing and to lessen the effects of small target size on acquisition difficulty, we introduce four enhanced area cursors, two of which rely on magnification and two of which use goal crossing. In a study with motor-impaired and able-bodied users, we compared the new designs to the point and Bubble cursors, the latter of which had not been evaluated for users with motor impairments. Two enhanced area cursors, the Visual-Motor-Magnifier and Click-and-Cross, were the most successful new designs for users with motor impairments, reducing selection time for small targets by 19%, corrective submovements by 45%, and error rate by up to 82% compared to the point cursor. Although the Bubble cursor also improved performance, participants with motor impairments unanimously preferred the enhanced area cursors.

Open bug reporting allows end-users to express a vast array of unwanted software behaviors. However, users’ expectations often clash with developers’ implementation intents. We created a classification of seven common expectation violations cited by end-users in bug report descriptions and applied it to 1,000 bug reports from the Mozilla project. Our results show that users largely described bugs as violations of their own personal expectations, of specifications, or of the user community’s expectations. We found a correlation between a reporter’s expression of which expectation was being violated and whether or not the bug would eventually be fixed. Specifically, when bugs were expressed as violations of community expectations rather than personal expectations, they had a better chance of being fixed.

Many dynamic programming language features, such as implicit declaration, reflection, and code generation, make it difficult to verify the existence of identifiers through standard program analysis. We present an alternative verification, which, rather than analyzing the semantics of code, highlights any name or pair of names that appear only once across a program's source files. This uniqueness heuristic is implemented for HTML, CSS, and JavaScript, in an interactive editor called Cleanroom, which highlights lone identifiers after each keystroke. Through an online experiment, we show that Cleanroom detects real errors, that it helps developers find these errors more quickly than developers can find them on their own, and that this helps developers avoid costly debugging effort by reducing how many times a program is executed with potential errors. The simplicity and power of Cleanroom's heuristic may generalize well to other dynamic languages with little support for edit-time name verification.

Mobile devices provide people with disabilities new opportunities to act independently in the world. However, these empowering devices have their own accessibility challenges. We present a formative study that examines how people with visual and motor disabilities select, adapt, and use mobile devices in their daily lives. We interviewed 20 participants with visual and motor disabilities and asked about their current use of mobile devices, including how they select them, how they use them while away from home, and how they adapt to accessibility challenges when on the go. Following the interviews, 19 participants completed a diary study in which they recorded their experiences using mobile devices for one week. Our results show that people with visual and motor disabilities use a variety of strategies to adapt inaccessible mobile devices and successfully use them to perform everyday tasks and navigate independently. We provide guidelines for more accessible and empowering mobile device design.

We conducted a 2.5 week longitudinal study with five motor impaired (MI) and four non-impaired (NMI) participants, in which they learned to use the Vocal Joystick, a voice-based user interface control system. We found that the participants were able to learn the mapping between the vowel sounds and directions used by the Vocal Joystick, and showed marked improvement in their target acquisition performance. At the end of the ten session period, the NMI group reached the same level of performance as the previously measured "expert" Vocal Joystick performance, and the MI group was able to reach 70% of that level. Two of the MI participants were also able to approach the performance of their preferred device, a touchpad. We report on a number of issues that can inform the development of further enhancements in the realm of voice-driven computer control.

We present a novel method of dynamic C-D gain adaptation that improves target acquisition for users with motor impairments. Our method, called the Angle Mouse, adjusts the mouse C-D gain based on the deviation of angles sampled during movement. When angular deviation is low, the gain is kept high. When angular deviation is high, the gain is dropped, making the target bigger in motor-space. A key feature of the Angle Mouse is that, unlike most pointing facilitation techniques, it is target-agnostic, requiring no knowledge of target locations or dimensions. This means that the problem of distractor targets is avoided because adaptation is based solely on the user's behavior. In a study of 16 people, 8 of which had motor impairments, we found that the Angle Mouse improved motor-impaired pointing throughput by 10.3% over the Windows default mouse and 11.0% over sticky icons. For able-bodied users, there was no significant difference among the three techniques, as Angle Mouse throughput was within 1.2% of the default. Thus, the Angle Mouse improved pointing performance for users with motor impairments while remaining unobtrusive for able-bodied users.

Many surface computing prototypes have employed gestures created by system designers. Although such gestures are appropriate for early investigations, they are not necessarily reflective of user behavior. We present an approach to designing tabletop gestures that relies on eliciting gestures from non-technical users by first portraying the effect of a gesture, and then asking users to perform its cause. In all, 1080 gestures from 20 participants were logged, analyzed, and paired with think-aloud data for 27 commands performed with 1 and 2 hands. Our findings indicate that users rarely care about the number of fingers they employ, that one hand is preferred to two, that desktop idioms strongly influence users' mental models, and that some commands elicit little gestural agreement, suggesting the need for on-screen widgets. We also present a complete user-defined gesture set, quantitative agreement scores, implications for surface technology, and a taxonomy of surface gestures. Our results will help designers create better gesture sets informed by user behavior.

We describe our system called MobileASL for real-time video communication on the current U.S. mobile phone network. The goal of MobileASL is to enable Deaf people to communicate with Sign Language over mobile phones by compressing and transmitting sign language video in real-time on an off-the-shelf mobile phone, which has a weak processor, uses limited bandwidth, and has little battery capacity. We develop several H.264-compliant algorithms to save system resources while maintaining ASL intelligibility by focusing on the important segments of the video. We employ a dynamic skin-based region-of-interest (ROI) that encodes the skin at higher quality at the expense of the rest of the video. We also automatically recognize periods of signing versus not signing and raise and lower the frame rate accordingly, a technique we call variable frame rate (VFR). We show that our variable frame rate technique results in a 47 % gain in battery life on the phone, corresponding to an extra 68 minutes of talk time. We also evaluate our system in a user study. Participants fluent in ASL engage in unconstrained conversations over mobile phones in a laboratory setting. We find that the ROI increases intelligibility and decreases guessing. VFR increases the need for signs to be repeated and the number of conversational breakdowns, but does not affect the users ’ perception of adopting the technology. These results show that our sign language sensitive algorithms can save considerable resources without sacrificing intelligibility. ACM Classification: H5.2 [Information interfaces and presentation]:Multimedia Information Systems–Video. K.4.2 [Computers and Society]: Social Issues–Assistive technologies for persons with disabilities. General terms:

We present Bonfire, a self-contained mobile computing system that uses two laptop-mounted laser micro-projectors to project an interactive display space to either side of a laptop keyboard. Coupled with each micro-projector is a camera to enable hand gesture tracking, object recognition, and information transfer within the projected space. Thus, Bonfire is neither a pure laptop system nor a pure tabletop system, but an integration of the two into one new nomadic computing platform. This integration (1) enables observing the periphery and responding appropriately, e.g., to the casual placement of objects within its field of view, (2) enables integration between physical and digital objects via computer vision, (3) provides a horizontal surface in tandem with the usual vertical laptop display, allowing direct pointing and gestures, and (4) enlarges the input/output space to enrich existing applications. We describe Bonfire's architecture, and offer scenarios that highlight Bonfire's advantages. We also include lessons learned and insights for further development and use.

TapSongs are presented, which enable user authentication on a single "binary" sensor (e.g., button) by matching the rhythm of tap down/up events to a jingle timing model created by the user. We describe our matching algorithm, which employs absolute match criteria and learns from successful logins. We also present a study of 10 subjects showing that after they created their own TapSong models from 12 examples (< 2 minutes), their subsequent login attempts were 83.2% successful. Furthermore, aural and visual eavesdropping of the experimenter's logins resulted in only 10.7% successful imposter logins by subjects. Even when subjects heard the target jingles played by a synthesized piano, they were only 19.4% successful logging in as imposters. These results are attributable to subtle but reliable individual differences in people's tapping, which are supported by prior findings in music psychology.

Recent advances in touch screen technology have increased the prevalence of touch screens and have prompted a wave of new touch screen-based devices. However, touch screens are still largely inaccessible to blind users, who must adopt error-prone compensatory strategies to use them or find accessible alternatives. This inaccessibility is due to interaction techniques that require the user to visually locate objects on the screen. To address this problem, we introduce Slide Rule, a set of audio-based multi-touch interaction techniques that enable blind users to access touch screen applications. We describe the design of Slide Rule, our interaction techniques, and a user study in which 10 blind people used Slide Rule and a button-based Pocket PC screen reader. Results show that Slide Rule was significantly faster than the button-based system, and was preferred by 7 of 10 users. However, users made more errors when using Slide Rule than when using the more familiar button-based system.

We evaluate two systems for automatically generating personalized interfaces adapted to the individual motor capabilities of users with motor impairments. The first system, SUPPLE, adapts to users' capabilities indirectly by first using the ARNAULD preference elicitation engine to model a user's preferences regarding how he or she likes the interfaces to be created. The second system, SUPPLE++, models a user's motor abilities directly from a set of one-time motor performance tests. In a study comparing these approaches to baseline interfaces, participants with motor impairments were 26.4% faster using ability-based user interfaces generated by SUPPLE++. They also made 73% fewer errors, strongly preferred those interfaces to the manufacturers' defaults, and found them more efficient, easier to use, and much less physically tiring. These findings indicate that rather than requiring some users with motor impairments to adapt themselves to software using separate assistive technologies, software can now adapt itself to the capabilities of its users.

For decades, Fitts' law (1954) has been used to model pointing time in user interfaces. As with any rapid motor act, faster pointing movements result in increased errors. But although prior work has examined accuracy as the "spread of hits," no work has formulated a predictive model for error rates (0-100%) based on Fitts' law parameters. We show that Fitts' law mathematically implies a predictive error rate model, which we derive. We then describe an experiment in which target size, target distance, and movement time are manipulated. Our results show a strong model fit: a regression analysis of observed vs. predicted error rates yields a correlation of R2=.959 for N=90 points. Furthermore, we show that the effect on error rate of target size (W) is greater than that of target distance (A), indicating a departure from Fitts' law, which maintains that W and A contribute proportionally to index of difficulty (ID). Our error model can be used with Fitts' law to estimate and predict error rates along with speeds, providing a framework for unifying this dichotomy.

Eye-typing performance results are reported from controlled studies comparing an on-screen keyboard and Eye Write, a new on-screen gestural input alternative. Results from the first pilot study suggest the presence of a learning curve that novice users must overcome in order to gain proficiency in EyeWrite's use (requiring practice with its letter-like gestural alphabet). Results from the second longitudinal study indicate that EyeWrite's inherent multi-saccade handicap (4.52 saccades per character, frequency-weighted average) is sufficient for the on-screen keyboard to edge out Eye Write in speed performance. Eye-typing speeds with Eye Write approach 5 wpm on average (8 wpm attainable by proficient users), whereas keyboard users achieve about 7 wpm on average (in line with previous results). However, Eye Write users leave significantly fewer uncorrected errors in the final text, with no significant difference in the number of errors corrected during entry, indicating a speed-accuracy trade-off. Subjective results indicate that participants consider Eye Write significantly faster, easier to use, and prone to cause less ocular fatigue than the on-screen keyboard. In addition, Eye-Write consumes much less screen real-estate than an on-screen keyboard, giving it practical advantages for eye-based text entry.

Many mobile machine learning applications require collecting and labeling data, and a traditional GUI on a mobile device may not be an appropriate or viable method for this task. This paper presents an alternative approach to mobile labeling of sensor data called VoiceLabel. VoiceLabel consists of two components: (1) a speech-based data collection tool for mobile devices, and (2) a desktop tool for offline segmentation of recorded data and recognition of spoken labels. The desktop tool automatically analyzes the audio stream to find and recognize spoken labels, and then presents a multimodal interface for reviewing and correcting data labels using a combination of the audio stream, the system's analysis of that audio, and the corresponding mobile sensor data. A study with ten participants showed that VoiceLabel is a viable method for labeling mobile sensor data. VoiceLabel also illustrates several key features that inform the design of other data labeling tools.

People with motor impairments often have difficulty typing using desktop keyboards. We developed TrueKeys, a system that combines models of word frequency, keyboard layout, and typing error patterns to automatically identify and correct typing mistakes. In this paper, we describe the TrueKeys algorithm, compare its performance to existing correction algorithms, and report on a study of TrueKeys with 9 motor-impaired and 9 non-impaired participants. Running in non-interactive mode, TrueKeys performed more corrections than popular commercial and open source spell checkers. Used interactively, both motor-impaired and non-impaired users performed typing tasks significantly more accurately with TrueKeys than without. However, typing speed was reduced while TrueKeys was enabled.

Using a mobile device while moving limits attention and motor ability and can result in reduced performance. Mobile devices that can sense and adapt to contextual factors such as movement may reduce this performance deficit. We performed two studies evaluating the feasibility of walking user interfaces (WUIs) that adapt their layout when the user is moving. In a pilot study with 6 users, we evaluated the effects of different button sizes on performance when walking while using a portable music player. Results showed significant interactions between size and movement. In the second study, 29 users evaluated the performance of a WUI that dynamically changed button sizes as the user moved. Results show that our dynamic user interface performs at the level of its component static interfaces without any additional penalty due to adaptation. This work adds to our design knowledge about walking user interfaces and provides lessons learned in evaluating mobile devices while walking in public spaces.

Web browsing is inefficient for blind web users because of persistent accessibility problems, but the extent of these problems and their practical effects from the perspective of the user has not been sufficiently examined. We conducted a study in situ to investigate the accessibility of the web as experienced by web users. This remote study used an advanced web proxy that leverages AJAX technology to record both the pages viewed and the actions taken by users on the web pages that they visited. Our study was conducted remotely over the period of one week, and our participants used the assistive technology and software to which they were already accustomed and had already configured according to preference. These advantages allowed us to aggregate observations of many users and to explore the practical effects on and coping strategies employed by our blind participants. Our study reflects web accessibility from the perspective of web users and describes quantitative differences in the browsing behavior of blind and sighted web users.

Mobile phones and personal digital assistants (PDAs) are incredibly popular pervasive technologies. Many of these devices contain touch screens, which can present problems for users with motor impairments due to small targets and their reliance on tapping for target acquisition. In order to select a target, users must tap on the screen, an action which requires the precise motion of flying into a target and lifting without slipping. In this paper, we propose a new technique for target acquisition called barrier pointing, which leverages the elevated physical edges surrounding the screen to improve pointing accuracy. After designing a series of barrier pointing techniques, we conducted an initial study with 9 able bodied users and 9 users with motor impairments in order to discover the parameters that make barrier pointing successful. From this data, we offer an in-depth analysis of the performance of two motor impaired users for whom barrier pointing was especially beneficial. We show the importance of providing physical stability by allowing the stylus to press against the screen and its physical edge. We offer other design insights and lessons learned that can inform future attempts at leveraging the physical properties of mobile devices to improve accessibility.

We present VoiceDraw, a voice-driven drawing application for people with motor impairments that provides a way to generate free-form drawings without needing manual interaction. VoiceDraw was designed and built to investigate the potential of the human voice as a modality to bring fluid, continuous direct manipulation interaction to users who lack the use of their hands. VoiceDraw also allows us to study the issues surrounding the design of a user interface optimized for non-speech voice-based interaction. We describe the features of the VoiceDraw application, our design process, including our user-centered design sessions with a 'voice painter', and offer lessons learned that could inform future voice-based design efforts. In particular, we offer insights for mapping human voice to continuous control.

Prior work has highlighted the challenges faced by people with motor impairments when trying to acquire on-screen targets using a mouse or trackball. Two reasons for this are the difficulty of positioning the mouse cursor within a confined area, and the challenge of accurately executing a click. We hypothesize that both of these difficulties with area pointing may be alleviated in a different target acquisition paradigm called "goal crossing." In goal crossing, users do not acquire a confined area, but instead pass over a target line. Although goal crossing has been studied for able-bodied users, its suitability for people with motor impairments is unknown. We present a study of 16 people, 8 of whom had motor impairments, using mice and trackballs to do area pointing and goal crossing. Our results indicate that Fitts' law models both techniques for both user groups. Furthermore, although throughput for able-bodied users was higher for area pointing than for goal crossing (4.72 vs. 3.61 bits/s), the opposite was true for users with motor impairments (2.34 vs. 2.88 bits/s), suggesting that goal crossing may be viable for them. However, error rates were higher for goal crossing than for area pointing under a strict definition of crossing errors (6.23% vs. 1.94%). Subjective results indicate a preference for goal crossing among motor-impaired users. This work provides the empirical foundation from which to pursue the design of crossing-based interfaces as accessible alternatives to pointing-based interfaces.

A gestural text entry method for mobile is presented. Unlike most mobile phone text entry methods, which rely on repeatedly pressing buttons, our gestural method uses an isometric joystick and the EdgeWrite alphabet to allow users to write by making letter-like "pressure strokes." In a 15-session study comparing character-level EdgeWrite to Multitap, subjects' speeds were statistically indistinguishable, reaching about 10 WPM. In a second 15-session study comparing word-level EdgeWrite to T9, the same subjects were again statistically indistinguishable, reaching about 16 WPM. Uncorrected errors were low, around 1% or less for each method. In addition, subjective results favored EdgeWrite. Overall, results indicate that our isometric joystick-based method is highly competitive with two commercial keypad-based methods, opening the way for keypad-less designs and text entry on tiny devices. Additional results showed that a joystick on the back could be used at about 70% of the speed of the front, and the front joystick could be used eyes-free at about 80% of the speed of normal use.

The increasing quantity and complexity of in-vehicle systems creates a demand for user interfaces which are suited to driving. The steering wheel is a common location for the placement of buttons to control navigation, entertainment, and environmental systems, but what about a small touchpad? To investigate this question, we embedded a Synaptics StampPad in a computer game steering wheel and evaluated seven methods for selecting from a list of over 3000 street names. Selection speed was measured while stationary and while driving a simulator. Results show that the EdgeWrite gestural text entry method is about 20% to 50% faster than selection-based text entry or direct list-selection methods. They also show that methods with slower selection speeds generally resulted in faster driving speeds. However, with EdgeWrite, participants were able to maintain their speed and avoid incidents while selecting and driving at the same time. Although an obvious choice for constrained input, on-screen keyboards generally performed quite poorly.

Most of today's GUIs are designed for the typical, able-bodied user; atypical users are, for the most part, left to adapt as best they can, perhaps using specialized assistive technologies as an aid. In this paper, we present an alternative approach: SUPPLE++ automatically generates interfaces which are tailored to an individual's motor capabilities and can be easily adjusted to accommodate varying vision capabilities. SUPPLE++ models users. motor capabilities based on a onetime motor performance test and uses this model in an optimization process, generating a personalized interface. A preliminary study indicates that while there is still room for improvement, SUPPLE++ allowed one user to complete tasks that she could not perform using a standard interface, while for the remaining users it resulted in an average time savings of 20%, ranging from an slowdown of 3% to a speedup of 43%.

Although mobile, tablet, large display, and tabletop computers increasingly present opportunities for using pen, finger, and wand gestures in user interfaces, implementing gesture recognition largely has been the privilege of pattern matching experts, not user interface prototypers. Although some user interface libraries and toolkits offer gesture recognizers, such infrastructure is often unavailable in design-oriented environments like Flash, scripting environments like JavaScript, or brand new off-desktop prototyping environments. To enable novice programmers to incorporate gestures into their UI prototypes, we present a "$1 recognizer" that is easy, cheap, and usable almost anywhere in about 100 lines of code. In a study comparing our $1 recognizer, Dynamic Time Warping, and the Rubine classifier on user-supplied gestures, we found that $1 obtains over 97% accuracy with only 1 loaded template and 99% accuracy with 3+ loaded templates. These results were nearly identical to DTW and superior to Rubine. In addition, we found that medium-speed gestures, in which users balanced speed and accuracy, were recognized better than slow or fast gestures for all three recognizers. We also discuss the effect that the number of templates or training examples has on recognition, the score falloff along recognizers' N-best lists, and results for individual gestures. We include detailed pseudocode of the $1 recognizer to aid development, inspection, extension, and testing.

We present a major extension to our previous work on Trackball EdgeWrite--a unistroke text entry method for trackballs--by taking it from a character-level technique to a word-level one. Our design is called stroke-based word completion, and it enables efficient word selection as part of the stroke-making process. Unlike most word completion designs, which require users to select words from a list, our technique allows users to select words by performing a fluid crossing gesture. Our theoretical model shows this word-level design to be 45.0% faster than our prior model for character-only strokes. A study with a subject with spinal cord injury comparing Trackball EdgeWrite to the onscreen keyboard WiViK, both using word prediction and completion, shows that Trackball EdgeWrite is competitive with WiViK in speed (12.09 vs. 11.82 WPM) and accuracy (3.95% vs. 2.21% total errors), but less visually tedious and ultimately preferred. The results also show that word-level Trackball EdgeWrite is 46.5% faster and 36.7% more accurate than our subject's prior peak performance with character-level Trackball EdgeWrite, and 75.2% faster and 40.2% more accurate than his prior peak performance with his preferred on-screen keyboard. An additional evaluation of the same subject over a two-month field deployment shows a 43.9% reduction in unistrokes due to strokebased word completion in Trackball EdgeWrite.

We present a new gestural text entry method for trackballs. The method uses the mouse cursor and relies on crossing instead of pointing. A user writes in fluid Roman-like unistrokes by ""pulsing"" the trackball in desired letter patterns. We examine this method both theoretically using the Steering Law and empirically in two studies. Our studies show that able-bodied users who were unfamiliar with trackballs could write at about 10 wpm with <4% total errors after 45 minutes. In eight sessions, a motor-impaired trackball user peaked at 7.11 wpm with 0% uncorrected errors, compared to 5.95 wpm with 0% uncorrected errors with an on-screen keyboard. Over sessions, his speeds were significantly faster with our gestural method than with an on-screen keyboard. A former 15-year veteran of on-screen keyboards, he now uses our gestural method instead.

We present a new 4-key text entry method that, unlike most few-key methods, is gestural instead of selection-based. Importantly, its gestures mimic the writing of Roman letters for high learnability. We compare this new 4-key method to predominant 3-key and 5-key methods theoretically using KSPC and empirically using a longitudinal study of 5 subjects over 10 sessions. The study includes an evaluation of the 4-key method without any on-screen visualization-an impossible condition for the selection-based methods. Our results show that the new 4-key method is quickly learned, becoming faster than the 3-key and 5-key methods after just ~10 minutes of writing, although it produces more errors. Interestingly, removing a visualization of the gestures being made causes no detriment to the 4-key method, which is an advantage for eyes-free text entry.

We present the design and implementation of a word-level stroking system called Fisch, which is intended to improve the speed of character-level unistrokes. Importantly, Fisch does not alter the way in which character-level unistrokes are made, but allows users to gradually ramp up to word-level unistrokes by extending their letters in minimal ways. Fisch relies on in-stroke word completion, a flexible design for fluidly turning unistroke letters into whole words. Fisch can be memorized at the motor level since word completions always appear at the same positions relative to the strokes being made. Our design for Fisch is suitable for use with any unistroke alphabet. We have implemented Fisch for multiple versions of EdgeWrite, and results show that Fisch reduces the number of strokes during entry by 43.9% while increasing the rate of entry. An informal test of "record speed" with the stylus version resulted in 50-60 wpm with no uncorrected errors.

Recent improvements in text entry error rate measurement have enabled the running of text entry experiments in which subjects are free to correct errors (or not) as they transcribe a presented string. In these “unconstrained” experiments, it is no longer necessary to force subjects to unnaturally maintain synchronicity with presented text for the sake of performing overall error rate calculations. However, the calculation of character-level error rates, which can be trivial in artificially constrained evaluations, is far more complicated in unconstrained text entry evaluations because it is difficult to infer a subject's intention at every character. For this reason, prior character-level error analyses for unconstrained experiments have only compared presented and transcribed strings, not input streams. But input streams are rich sources of character-level error information, since they contain all of the text entered (and erased) by a subject. The current work presents an algorithm for the automated analysis of character-level errors in input streams for unconstrained text entry evaluations. It also presents new character-level metrics that can aid method designers in refining text entry methods. To exercise these metrics, we perform two analyses on data from an actual text entry experiment. One analysis, available from the prior work, uses only presented and transcribed strings. The other analysis uses input streams, as described in the current work. The results confirm that input stream error analysis yields richer information for the same empirical data. To facilitate the use of these new analyses, we offer pseudocode and downloadable software for performing unconstrained text entry experiments and analyzing data.

We present various adaptations of the EdgeWrite unistroke text entry method that work on multiple computer input devices: styluses, touchpads, displacement and isometric joysticks, four keys or buttons, and trackballs. We argue that consistent, flexible, multi-device input is important to both accessibility and to ubiquitous computing. For accessibility, multi-device input means users can switch among devices, distributing strain and fatigue among different muscle groups. For ubiquity, it means users can "learn once, write anywhere," even as new devices emerge. By considering the accessibility and ubiquity of input techniques, we can design for both motor-impaired users and "situationally impaired" able-bodied users who are on-the-go. We discuss the requirements for such input and the challenges of multi-device text entry, such as solving the segmentation problem. This paper accompanies a demonstration of EdgeWrite on multiple devices.

Power wheelchair joysticks have be used to control a mouse cursor on desktop computers, but they offer no integrated text entry solution, confining users to point-and-click or point-and-dwell with on-screen keyboards. On-screen keyboards reduce useful screen real-estate, exacerbating the need for frequent window management, and impose a secondary focus of attention. By contrast, we present two integrated gestural text entry methods designed for use from power wheelchairs: one for use with joysticks and the other for use with touchpads. Both techniques are adaptations of EdgeWrite, originally a stylus-based unistroke method designed for people with tremor. In a preliminary text entry study of 7 power wheelchair users, we found that EdgeWrite with a touchpad was faster than the on-screen keyboard WiViK with a joystick, and EdgeWrite with a joystick was only slightly slower. These results warranted a multi-session comparison of text entry with EdgeWrite and WiViK using joysticks and touchpads, in which we found touchpads faster than joysticks, and EdgeWrite faster than WiViK with both devices after initial learning periods.

Power wheelchair joysticks have been used to control a mouse cursor on desktop computers, but they offer no integrated text entry solution, confining users to point-and-click or point-and-dwell with on-screen keyboards. But on-screen keyboards reduce useful screen real-estate, exacerbate the need for frequent window management, and impose a second focus of attention. By contrast, we present two integrated gestural text entry methods designed for use from power wheelchairs: one for joysticks and the other for touchpads. Both techniques are adaptations of EdgeWrite, originally a stylus-based unistroke method designed for people with tremor. In a preliminary study of 7 power wheelchair users, we found that touchpad EdgeWrite was faster than joystick WiVik, and joystick EdgeWrite was only slightly slower after minimal practice. These findings reflect "walk up and use"-ability and warrant further investigation into extended use.

EdgeWrite is a new unistroke text entry method for handheld devices designed to provide high accuracy and stability of motion for people with motor impairments. It is also effective for able-bodied people. An EdgeWrite user enters text by traversing the edges and diagonals of a square hole imposed over the usual text input area. Gesture recognition is accomplished not through pattern recognition but through the sequence of corners that are hit. This means that the full stroke path is unimportant and recognition is highly deterministic, enabling better accuracy than other gestural alphabets such as Graffiti. A study of able-bodied users showed subjects with no prior experience were 18% more accurate during text entry with Edge Write than with Graffiti (p>.05), with no significant difference in speed. A study of 4 subjects with motor impairments revealed that some of them were unable to do Graffiti, but all of them could do Edge Write. Those who could do both methods had dramatically better accuracy with Edge Write.

People with Muscular Dystrophy (MD) and certain other muscular and nervous system disorders lose their gross motor control while retaining fine motor control. The result is that they lose the ability to move their wrists and arms, and therefore their ability to operate a mouse and keyboard. However, they can often still use their fingers to control a pencil or stylus, and thus can use a handheld computer such as a Palm. We have developed software that allows the handheld to substitute for the mouse and keyboard of a PC, and tested it with four people (ages 10, 12, 27 and 53) with MD. The 12-year old had lost the ability to use a mouse and keyboard, but with our software, he was able to use the Palm to access email, the web and computer games. The 27-year-old reported that he found the Palm so much better that he was using it full-time instead of a keyboard and mouse. The other two subjects said that our software was much less tiring than using the conventional input devices, and enabled them to use computers for longer periods. We report the results of these case studies, and the adaptations made to our software for people with disabilities.

The proliferation of wireless handheld devices is placing the World Wide Web in the palms of users, but this convenience comes at a high interactive cost. The Web that came of age on the desktop is ill-suited for use on the small displays of handhelds. Today, handheld browsing often feels like browsing on a PC with a shrunken desktop. Overreliance on scrolling is a big problem in current handheld browsing. Users confined to viewing a small portion of each page often lack a sense of the overall context --- they may feel lost in a large page and be forced to remember the locations of items as those items scroll out of view. In this paper, we present a synthesis of interaction techniques to address these problems. We implemented these techniques in a prototype, WebThumb, that can browse the live Web.