Accessibility checkers are tools in support of accessible app development, and their use is encouraged by accessibility best practices. However, most current checkers evaluate static or mechanically-generated contexts, failing to capture common accessibility errors impacting mobile app functionality. In this work, we define functiona11ity errors as accessibility barriers that only manifest through interaction (i.e., named according to a blend of "functionality" and "accessibility"). We introduce TaskAudit, which comprises three components: a Task Generator that constructs interactive tasks from app screens, a Task Executor that uses agents with a screen reader proxy to perform these tasks, and an Accessibility Analyzer that detects and reports accessibility errors by examining interaction traces. Our evaluation on real-world apps shows that TaskAudit detects 48 functiona11ity errors from 54 app screens, compared to between 4 and 20 with existing checkers. Our analysis demonstrates common error patterns that TaskAudit can detect in addition to those from prior work, including label-functionality mismatch, cluttered navigation, and inappropriate feedback.

Accessibility research has a broad mandate: use technology to make the world more accessible to disabled people. Yet, as a field, accessibility research lacks a clear characterization of what "accessibility" is. Furthermore, it has been historically limited in who is designed for, focusing on specific types of disability and often failing to consider how disability intersects with other identities. We set out to explicate what it means to make something accessible, grounded in the lived experiences of a diverse group of 25 disabled people. From our empirical findings, we develop a process for modeling accessibility. First, an individual assesses their experience of inaccess, specifically, the type of barrier they face, the technology repertoire they possess, and the contextual factors that shape how they address accessibility barriers. Then, having assessed an access barrier, they perform consequence calculus, weighing all available options to achieve access and deciding upon the option that best matches their priorities. We highlight the situated nature of access; people's identities, contextual factors, repertoires, and priorities all dictate their experience of accessibility.

Many mobile apps are inaccessible, thereby excluding people from their potential benefits. Existing rule-based accessibility checkers aim to mitigate these failures by identifying errors early during development but are constrained in the types of errors they can detect. We present ScreenAudit, an LLM-powered system designed to traverse mobile app screens, extract metadata and transcripts, and identify screen reader accessibility errors overlooked by existing checkers. We recruited six accessibility experts including one screen reader user to evaluate ScreenAudit's reports across 14 unique app screens. Our findings indicate that ScreenAudit achieves an average coverage of 69.2%, compared to only 31.3% with a widely-used accessibility checker. Expert feedback indicated that ScreenAudit delivered higher-quality feedback and addressed more aspects of screen reader accessibility compared to existing checkers, and that ScreenAudit would benefit app developers in real-world settings.

Deceptive design patterns manipulate people into actions to which they would otherwise object. Despite growing research on deceptive design patterns, limited research examines their interplay with accessibility and visual accessibility technology (e.g., screen readers, screen magnification, braille displays). We present an interview and diary study with 16 people who use visual accessibility technology to better understand experiences with accessibility and deceptive design. We report participant experiences with six deceptive design patterns, including designs that are intentionally deceptive and designs where participants describe accessibility barriers unintentionally manifesting as deceptive, together with direct and indirect consequences of deceptive patterns. We discuss intent versus impact in accessibility and deceptive design, how access barriers exacerbate harms of deceptive design patterns, and impacts of deceptive design from a perspective of consequence-based accessibility. We propose that accessibility tools could help address deceptive design patterns by offering higher-level feedback to well-intentioned designers.

Chronic liver disease can lead to neurological conditions that result in coma or death. Although early detection can allow for intervention, testing is infrequent and unstandardized. Beacon is a device for at-home patient self-measurement of cognitive function via critical flicker frequency, which is the frequency at which a flickering light appears steady to an observer. This paper presents our efforts in iterating on Beacon's hardware and software to enable at-home use, then reports on an at-home deployment with 21 patients taking measurements over 6 weeks. We found that measurements were stable despite being taken at different times and in different environments. Finally, through interviews with 15 patients and 5 hepatologists, we report on participant experiences with Beacon, preferences around how CFF data should be presented, and the role of caregivers in helping patients manage their condition. Informed by our experiences with Beacon, we further discuss design implications for home health devices.

Collaborative care management is an evidence-based approach to integrated psychosocial care for patients with comorbid cancer and depression. Prior work highlights challenges in patient-provider collaboration in navigating parallel cancer care and psychosocial care journeys of these patients. We design and deploy SCOPE , a platform for technology-enhanced collaborative care combining a patient-facing mobile app with a provider-facing registry. We examine SCOPE through a total of 45 interviews with patients and providers conducted in SCOPE 's 15 months of design and development and 24 Months of SCOPE 's deployment for actual care in 6 cancer clinics. We find that: (1) SCOPE supported patient engagement in its underlying collaborative care and behavioral activation interventions, (2) patient-generated data in SCOPE improved patient-provider collaboration between and within in-person sessions, (3) SCOPE supported providers in delivering care and improved care team collaboration, (4) experience with SCOPE created evolving expectations for collaboration around data, and (5) SCOPE 's deployment in actual care surfaced important implementation barriers. We discuss the implications of our findings in terms of designing for engagement with behavioral health interventions, negotiating patient data sharing and provider responsiveness, supporting personalized self-tracking goals in evidence-based interventions, exploring the role of digital health navigators in technology-enhanced care, and the need for flexibility in aligning technology-supported interventions to patient needs.

Video games often pose accessibility barriers to gamers with disabilities, yet there is no standard method for identifying which games have barriers, what those barriers are, and whether and how they can be overcome. We propose and explore three phases of the “game adoption process”: Discovery, Evaluation, and Adaptation. To advance understanding of how gamers with disabilities experience this process, the resources and strategies they use, and the challenges experienced, we conducted an interview study with thirteen gamers with disabilities with differing backgrounds. We then engage with existing theories of consequence-based accessibility, of difficulty, and of identity-based gaming to better understand how these processes manifest “access difficulty” and to characterize the experience of “disabled gaming.” Finally, we present design recommendations for game developers and distributors to better support gamers with disabilities in the game adoption process by engaging with community-made resources, supporting socially-created access, and creating customizable experiences with opportunities for unconventional play.

Self-tracking and personal informatics offer important potential in chronic condition management, but such potential is often undermined by difficulty in aligning self-tracking tools to an individual's goals. Informed by prior proposals of goal-directed tracking, we designed and developed MigraineTracker, a prototype app that emphasizes explicit expression of goals for migraine-related self-tracking. We then examined migraine patient experiences in a deployment study for an average of 12+ months, including a total of 50 interview sessions with 10 patients working with 3 different clinicians. Patients were able to express multiple types of goals, evolve their goals over time, align tracking to their goals, personalize their tracking, reflect in the context of their goals, and gain insights that enabled understanding, communication, and action. We discuss how these results highlight the importance of accounting for distinct and concurrent goals in personal informatics together with implications for the design of future goal-directed personal informatics tools.

Menopause is often overlooked or medicalized, consequently devaluing individual experiences and failing to support individuals experiencing this life event. Family dynamics, death, and taboo further mean that individuals often miss out on information that could help them contextualize their experiences. We examine participant experiences with menopause and explore designs of digital and non-digital legacies for sharing menopause experiences across generations. We conducted semi-structured interviews and design sessions with 17 participants who experienced or are experiencing menopause. We report participant information needs and sense-making practices, including what personalized information participants wish to pass down and preferred formats for intergenerational sharing. Findings highlight the potential of using storytelling and life-logging to create "holistic" memories of the menopause journey, to support self-reflection, and for using legacies to initiate conversations about marginalized health experiences. We identify future design and research opportunities for the HCI and CSCW communities to support intergenerational sharing of non-medicalized and stigmatized health experiences.

Despite significant progress in the capabilities of mobile devices and applications, most apps remain oblivious to their users' abilities. To enable apps to respond to users' situated abilities, we created the Ability-Based Design Mobile Toolkit (ABD-MT). ABD-MT integrates with an app's user input and sensors to observe a user's touches, gestures, physical activities, and attention at runtime, to measure and model these abilities, and to adapt interfaces accordingly. Conceptually, ABD-MT enables developers to engage with a user's "ability profile,'' which is built up over time and inspectable through our API. As validation, we created example apps to demonstrate ABD-MT, enabling ability-aware functionality in 91.5% fewer lines of code compared to not using our toolkit. Further, in a study with 11 Android developers, we showed that ABD-MT is easy to learn and use, is welcomed for future use, and is applicable to a variety of end-user scenarios.

Videoconferencing usage has surged in recent years, but current platforms present significant accessibility barriers for the 430 million d/Deaf or hard of hearing people worldwide. Informed by prior work examining accessibility barriers in current videoconferencing platforms, we designed and developed Jod, a videoconferencing platform to facilitate communication in mixed hearing groups. Key features include support for customizing visual layouts and a notification system to request attention and influence behavior. Using Jod, we conducted six mixed hearing group sessions with 34 participants, including 18 d/Deaf or hard of hearing participants, 10 hearing participants, and 6 sign language interpreters. We found participants engaged in visual layout rearrangements based on their hearing ability and dynamically adapted to the changing group communication context, and that notifications were useful but raised a need for designs to cause fewer interruptions. We provide insights for future videoconferencing designs and conclude with recommendations for conducting mixed hearing studies.

Measures of human performance for touch-based systems have focused mainly on overall metrics like touch accuracy and target acquisition speed. But touches are not atomic—they unfold over time and space, especially for users with limited fine motor function, for whom it can be difficult to perform quick, accurate touches. To gain insight into what happens during a touch, we offer 15 target-agnostic touch metrics, most of which have not been mathematically formalized in the literature. They are touch direction, variability, drift, duration, extent, absolute/signed area change, area variability, area deviation, area extent, absolute/signed angle change, angle variability, angle deviation, and angle extent. These metrics regard a touch as a time series of ovals instead of a mere (x, y) coordinate. We provide mathematical definitions and visual depictions of our metrics, and consider policies for calculating our metrics when multiple fingers perform coincident touches. To exercise our metrics, we collected touch data from 27 participants, 15 of whom reported having limited fine motor function. Our results show that our metrics effectively characterize touch behaviors including fine-motor challenges. Our metrics can be useful for both understanding users and for evaluating touch-based systems to inform their design.

We present the first large-scale longitudinal analysis of missing label accessibility failures in Android apps. We developed a crawler and collected monthly snapshots of 312 apps over 16 months. We use this unique dataset in empirical examinations of accessibility not possible in prior datasets. Key large-scale findings include missing label failures in 55.6% of unique image-based elements, longitudinal improvement in ImageButton elements but not in more prevalent ImageView elements, that 8.8% of unique screens are unreachable without navigating at least one missing label failure, that app failure rate does not improve with number of downloads, and that effective labeling is neither limited to nor guaranteed by large software organizations. We then examine longitudinal data in individual apps, presenting illustrative examples of accessibility impacts of systematic improvements, incomplete improvements, interface redesigns, and accessibility regressions. We discuss these findings and potential opportunities for tools and practices to improve label-based accessibility.

Current performance measures with touch-based systems usually focus on overall performance, such as touch accuracy and target acquisition speed. But a touch is not an atomic event; it is a process that unfolds over time, and this process can be characterized to gain insight into users’ touch behaviors. To this end, our work proposes 13 target-agnostic touch performance metrics to characterize what happens during a touch. These metrics are: touch direction, variability, drift, duration, extent, absolute/signed area change, area variability, area deviation, absolute/signed angle change, angle variability, and angle deviation. Unlike traditional touch performance measures that treat a touch as a single (x, y) coordinate, we regard a touch as a time series of ovals that occur from finger-down to finger-up. We provide a mathematical formula and intuitive description for each metric we propose. To evaluate our metrics, we run an analysis on a publicly available dataset containing touch inputs by people with and without limited fine motor function, finding our metrics helpful in characterizing different fine motor control challenges. Our metrics can be useful to designers and evaluators of touch-based systems, particularly when making touch screens accessible to all forms of touch.

Android educational games are powerful learning tools but small, moving targets and game implementations pose accessibility challenges to people with upper-body motor impairments. In this poster, we present findings from a qualitative accessibility evaluation of 30 popular Android educational games, identify and reflect on accessibility barriers, and provide preliminary design recommendations.

Personal health informatics continues to grow in both research and practice, revealing many challenges of designing applications that address people's needs in their health, everyday lives, and collaborations with clinicians. Research suggests strategies to address such challenges, but has struggled to translate these strategies into design practice. This study examines translation of insights from personal health informatics research into resources to support designers. Informed by a review of relevant literature, we present our development of a prototype set of design cards intended to support designers in re-thinking potential assumptions about personal health informatics. We examined our design cards in semi-structured interviews, first with 12 student designers and then with 12 health-focused professional designers and researchers. Our results and discussion reveal tensions and barriers designers encounter, the potential for translational resources to inform the design of health-related technologies, and a need to support designers in addressing challenges of knowledge, advocacy, and evidence in designing for health.

Although exploring alternatives is fundamental to creating better interface designs, current processes for creating alternatives are generally manual, limiting the alternatives a designer can explore. We present Scout, a system that helps designers rapidly explore alternatives through mixed-initiative interaction with high-level constraints and design feedback. Prior constraint-based layout systems use low-level spatial constraints and generally produce a single design. Tosupport designer exploration of alternatives, Scout introduces high-level constraints based on design concepts (e.g.,~semantic structure, emphasis, order) and formalizes them into low-level spatial constraints that a solver uses to generate potential layouts. In an evaluation with 18 interface designers, we found that Scout: (1) helps designers create more spatially diverse layouts with similar quality to those created with a baseline tool and (2) can help designers avoid a linear design process and quickly ideate layouts they do not believe they would have thought of on their own.

People often do not receive the reactions they desire when they use social networking sites to share data collected through personal tracking tools like Fitbit, Strava, and Swarm. Although some people have found success sharing with close connections or in finding online communities, most audiences express limited interest and rarely respond. We report on findings from a human-centered design process undertaken to examine how tracking tools can better support people in telling their story using their data formative interviews contribute design goals for telling stories of accomplishment, including a need to include relevant data. We implement these goals in Yarn, a mobile app that offers structure for telling stories of accomplishment around training for running races and completing Do-It-Yourself projects.1 participants used Yarn for 4 weeks across two studies. Although Yarn's structure led some participants to include more data or explanation in the moments they created, many felt like the structure prevented them from telling their stories in the way they desired. In light of participant use, we discuss additional challenges to using personal data to inform and target an interested audience.

Parents and their school-age children can impact one another's sleep. Most sleep-tracking tools, however, are designed for adults and make it difficult for parents and children to track together. To examine how to design a family-centered sleep tracking tool, we designed DreamCatcher. DreamCatcher is an in-home, interactive, shared display that aggregates data from wrist-worn sleep sensors and self-reported mood. We deployed DreamCatcher as a probe to examine the design space of tracking sleep as a family. Ten families participated in the study probe between 15 and 50 days. This study uses a family systems perspective to explore research questions regarding the feasibility of children actively tracking health data alongside their parents and the effects of tracking and sharing on family dynamics. Our results indicate that children can be active tracking contributors and that having parents and children track together encourages turn-taking and working together. However, there were also moments when family members, in particular parents, felt discomfort from sharing their sleep and mood with other family members. Our research contributes to a growing understanding of designing family-centered health-informatics tools to support the combined needs of parents and children.

Depression is common but under-treated in patients with cancer, despite being a major modifiable contributor to morbidity and early mortality. Integrating psychosocial care into cancer services through the team-based Collaborative Care Management (CoCM) model has been proven to be effective in improving patient outcomes in cancer centers. However, there is currently a gap in understanding the challenges that patients and their care team encounter in managing co-morbid cancer and depression in integrated psycho-oncology care settings. Our formative study examines the challenges and needs of CoCM in cancer settings with perspectives from patients, care managers, oncologists, psychiatrists, and administrators, with a focus on technology opportunities to support CoCM. We find that: (1) patients with co-morbid cancer and depression struggle to navigate between their cancer and psychosocial care journeys, and (2) conceptualizing co-morbidities as separate and independent care journeys is insufficient for characterizing this complex care context. We then propose the parallel journeys framework as a conceptual design framework for characterizing challenges that patients and their care team encounter when cancer and psychosocial care journeys interact. We use the challenges discovered through the lens of this framework to highlight and prioritize technology design opportunities for supporting whole-person care for patients with co-morbid cancer and depression.

A digital calculator is one of the most frequently used touch screen applications. However, keypad-based character input in existing calculator applications requires precise, targeted key presses that are time-consuming and error-prone for many screen readers users. We introduce GestureCalc, a digital calculator that uses target-free gestures for arithmetic tasks. It allows eyes-free target-less input of digits and operations through taps and directional swipes with one to three fingers, guided by minimal audio feedback. We conducted a mixed methods longitudinal study with eight screen reader users and found that they entered characters with GestureCalc 40.5% faster on average than with a typical touch screen calculator. Participants made more mistakes but also corrected more errors with GestureCalc, resulting in 52.2% fewer erroneous calculations than the baseline. Over the three sessions in the study, participants were able to learn the GestureCalc gestures and efficiently perform short calculations. From our interviews after the second session, participants recognized the effort in learning a new gesture set, yet reported confidence in their ability to become fluent in practice.

Keypad-based character input in existing digital calculator applications on touch screen devices requires precise, targeted key presses that are time-consuming and error-prone for many screen reader users. We demonstrate GestureCalc, a digital calculator that uses target-free gestures for arithmetic tasks. It allows eyes-free target-less input of digits and operations through taps and directional swipes with one to three fingers, guided by minimal audio feedback. A study of the effectiveness of GestureCalc for screen reader users appears in a full paper by the authors at this conference.

Self-tracked health data can help people and their health providers understand and manage chronic conditions. This paper examines personal informatics practices and challenges in migraine, a condition characterized by unpredictable, intermittent, and poorly-understood symptoms. To investigate how people with migraine track and use data related to their condition, we surveyed 279 people with migraine and conducted semi-structured interviews with 13 survey respondents and 6 health providers. We find four distinct goals people bring to tracking and data: 1) answering questions about migraines, 2) predicting and preventing migraines, 3) monitoring and managing migraines over time, and 4) enabling motivation and social recognition. Each goal suggests different needs for the design of tools to support migraine tracking. We also find needs resulting from an individual's goals evolving over time, their varied personal experiences, and their communication and collaboration with providers. We discuss these goals and needs in terms of opportunities for personal informatics tools to facilitate learning to: 1) avoid common pitfalls; 2) support customization and flexibility; 3) account for burden, negativity, and lapsing; and 4) support management with uncertainty.

We conduct the first large-scale analysis of the accessibility of mobile apps, examining what unique insights this can provide into the state of mobile app accessibility. We analyzed 5,753 free Android apps for label-based accessibility barriers in three classes of image-based buttons: Clickable Images, Image Buttons, and Floating Action Buttons. An epidemiology-inspired framework was used to structure the investigation. The population of free Android apps was assessed for label-based inaccessible button diseases. Three determinants of the disease were considered: missing labels, duplicate labels, and uninformative labels. The prevalence, or frequency of occurrences of barriers, was examined in apps and in classes of image-based buttons. In the app analysis, 35.9% of analyzed apps had 90% or more of their assessed image-based buttons labeled, 45.9% had less than 10% of assessed image-based buttons labeled, and the remaining apps were relatively uniformly distributed along the proportion of elements that were labeled. In the class analysis, 92.0% of Floating Action Buttons were found to have missing labels, compared to 54.7% of Image Buttons and 86.3% of Clickable Images. We discuss how these accessibility barriers are addressed in existing treatments, including accessibility development guidelines.

The absence of tactile cues such as keys and buttons makes touchscreens difficult to navigate for people with visual impairments. Increasing tactile feedback and tangible interaction on touchscreens can improve their accessibility. However, prior solutions have either required hardware customization or provided limited functionality with static overlays. Prior investigation of tactile solutions for large touchscreens also may not address the challenges on mobile devices. We therefore present Interactiles, a low cost, portable, and unpowered system that enhances tactile interaction on Android touchscreen phones. Interactiles consists of 3D-printed hardware interfaces and software that maps interaction with that hardware to manipulation of a mobile app. The system is compatible with the built-in screen reader without requiring modification of existing mobile apps. We describe the design and implementation of Interactiles, and we evaluate its improvement in task performance and the user experience it enables with people who are blind or have low vision.

Accessibility issues in mobile apps make those apps difficult or impossible to access for many people. Examples include elements that fail to provide alternative text for a screen reader, navigation orders that are difficult, or custom widgets that leave key functionality inaccessible. Social annotation techniques have demonstrated compelling approaches to such accessibility concerns in the web, but have been difficult to apply in mobile apps because of the challenges of robustly annotating interfaces. This research develops methods for robust annotation of mobile app interface elements. Designed for use in runtime interface modification, our methods are based in screen identifiers, element identifiers, and screen equivalence heuristics. We implement initial developer tools for annotating mobile app accessibility metadata, evaluate our current screen equivalence heuristics in a dataset of 2038 screens collected from 50 mobile apps, present three case studies implementing runtime repair of common accessibility issues, and examine repair of real-world accessibility issues in 26 apps. These contributions overall demonstrate strong opportunities for social annotation in mobile accessibility.

Mobile accessibility is often a property considered at the level of a single mobile application (app), but rarely on a larger scale of the entire app "ecosystem," such as all apps in an app store, their companies, developers, and user influences. We present a novel conceptual framework for the accessibility of mobile apps inspired by epidemiology. It considers apps within their ecosystems, over time, and at a population level. Under this metaphor, "inaccessibility" is a set of diseases that can be viewed through an epidemiological lens. Accordingly, our framework puts forth notions like risk and protective factors, prevalence, and health indicators found within a population of apps. This new framing offers terminology, motivation, and techniques to reframe how we approach and measure app accessibility. It establishes how app accessibility can benefit from multi-factor, longitudinal, and population-based analyses. Our epidemiology-inspired conceptual framework is the main contribution of this work, intended to provoke thought and inspire new work enhancing app accessibility at a systemic level. In a preliminary exercising of our framework, we perform an analysis of the prevalence of common determinants or accessibility barriers. We assess the health of a stratified sample of 100 popular Android apps using Google's Accessibility Scanner. We find that 100% of apps have at least one of nine accessibility errors and examine which errors are most common. A preliminary analysis of the frequency of co-occurrences of multiple errors in a single app is also presented. We find 72% of apps have five or six errors, suggesting an interaction among different errors or an underlying influence.

Many people appropriate social media and online communities in their pursuit of personal health goals, such as healthy eating or increased physical activity. However, people struggle with impression management, and with reaching the right audiences when they share health information on these platforms. Instagram, a popular photo-based social media platform, has attracted many people who post and share their food photos. We aim to inform the design of tools to support healthy behaviors by understanding how people appropriate Instagram to track and share food data, the benefits they obtain from doing so, and the challenges they encounter. We interviewed 16 women who consistently record and share what they eat on Instagram. Participants tracked to support themselves and others in their pursuit of healthy eating goals. They sought social support for their own tracking and healthy behaviors and strove to provide that support for others. People adapted their personal tracking practices to better receive and give this support. Applying these results to the design of health tracking tools has the potential to help people better access social support.

We consider why and how women track their menstrual cycles, examining their experiences to uncover design opportunities and extend the field's understanding of personal informatics tools. To understand menstrual cycle tracking practices, we collected and analyzed data from three sources: 2,000 reviews of popular menstrual tracking apps, a survey of 687 people, and follow-up interviews with 12 survey respondents. We find that women track their menstrual cycle for varied reasons that include remembering and predicting their period as well as informing conversations with healthcare providers. Participants described six methods of tracking their menstrual cycles, including use of technology, awareness of their premenstrual physiological states, and simply remembering. Although women find apps and calendars helpful, these methods are ineffective when predictions of future menstrual cycles are inaccurate. Designs can create feelings of exclusion for gender and sexual minorities. Existing apps also generally fail to consider life stages that women experience, including young adulthood, pregnancy, and menopause. Our findings encourage expanding the field's conceptions of personal informatics.

We present Group Touch, a method for distinguishing among multiple users simultaneously interacting with a tabletop computer using only the touch information supplied by the device. Rather than tracking individual users for the duration of an activity, Group Touch distinguishes users from each other by modeling whether an interaction with the tabletop corresponds to either: (1) a new user, or (2) a change in users currently interacting with the tabletop. This reframing of the challenge as distinguishing users rather than tracking and identifying them allows Group Touch to support multi-user collaboration in real-world settings without custom instrumentation. Specifically, Group Touch examines pairs of touches and uses the difference in orientation, distance, and time between two touches to determine whether the same person performed both touches in the pair. Validated with field data from high-school students in a classroom setting, Group Touch distinguishes among users "in the wild" with a mean accuracy of 92.92% (SD=3.94%). Group Touch can imbue collaborative touch applications in real-world settings with the ability to distinguish among multiple users.

Diagnostic self-tracking, the recording of personal information to diagnose or manage a health condition, is a common practice, especially for people with chronic conditions. Unfortunately, many who attempt diagnostic self-tracking have trouble accomplishing their goals. People often lack knowledge and skills needed to design and conduct scientifically rigorous experiments, and current tools provide little support. To address these shortcomings and explore opportunities for diagnostic self-tracking, we designed, developed, and evaluated a mobile app that applies a self-experimentation framework to support patients suffering from irritable bowel syndrome (IBS) in identifying their personal food triggers. TummyTrials aids a person in designing, executing, and analyzing self-experiments to evaluate whether a specific food triggers their symptoms. We examined the feasibility of this approach in a field study with 15 IBS patients, finding that participants could use the tool to reliably undergo a self-experiment. However, we also discovered an underlying tension between scientific validity and the lived experience of self-experimentation. We discuss challenges of applying clinical research methods in everyday life, motivating a need for the design of self-experimentation systems to balance rigor with the uncertainties of everyday life.

Most web applications are designed as one-size-fits-all, despite considerable variation in people's expertise, physical abilities, and other factors that impact interaction. For example, some web applications require the use of a mouse, precluding use by many people with severe motor disabilities. Other applications require laborious manual input that a skilled developer could automate if the application were scriptable. This paper presents Genie, a system that automatically reverse engineers an abstract model of the underlying commands in a web application, then enables interaction with that functionality through alternative interfaces and other input modalities (e.g., speech, keyboard, or command line input). Genie comprises an abstract model of command properties, behaviors, and dependencies as well as algorithms that reverse engineer this model from an existing web application through static and dynamic program analysis. We evaluate Genie by developing several interfaces that automatically add support for speech, keyboard, and command line input to arbitrary web applications.

We introduce interaction proxies as a strategy for runtime repair and enhancement of the accessibility of mobile applications. Conceptually, interaction proxies are inserted between an application's original interface and the manifest interface that a person uses to perceive and manipulate the application. This strategy allows third-party developers and researchers to modify an interaction without an application's source code, without rooting the phone, without otherwise modifying an application, while retaining all capabilities of the system (e.g., Android's full implementation of the TalkBack screen reader). This paper introduces interaction proxies, defines a design space of interaction re-mappings, identifies necessary implementation abstractions, presents details of implementing those abstractions in Android, and demonstrates a set of Android implementations of interaction proxies from throughout our design space. We then present a set of interviews with blind and low-vision people interacting with our prototype interaction proxies, using these interviews to explore the seamlessness of interaction, the perceived usefulness and potential of interaction proxies, and visions of how such enhancements could gain broad usage. By allowing third-party developers and researchers to improve an interaction, interaction proxies offer a new approach to personalizing mobile application accessibility and a new approach to catalyzing development, deployment, and evaluation of mobile accessibility enhancements.

In families composed of parents and children, the health of parents and children is often interrelated: the health of children can have an impact on the health of parents, and vice versa. However, the design of health tracking technologies typically focuses on individual self-tracking and self-management, not yet addressing family health in a unified way. To examine opportunities for family-centered health informatics, we interviewed 14 typically healthy families, interviewed 10 families with a child with a chronic condition, and conducted three participatory design sessions with children aged 7 to 11. Although we identified similarities between family-centered tracking and personal self-tracking, we also found families want to: (1) identify ripple effects between family members; (2) consider both caregivers and children as trackers to support distributing the burdens of tracking across family members; and (3) identify and pursue health guidelines that consider the state of their family (e.g., specific health guidelines for families that include a child with a chronic condition). We contribute to expanding the design lens from self-tracking to family-centered health tracking.

Patient-generated data can allow patients and providers to collaboratively develop accurate diagnoses and actionable treatment plans. Unfortunately, patients and providers often lack effective support to make use of such data. We examine patient-provider collaboration to interpret patient-generated data. We focus on irritable bowel syndrome (IBS), a chronic illness in which particular foods can exacerbate symptoms. IBS management often requires patient-provider collaboration using a patient's food and symptom journal to identify the patient's triggers. We contribute interactive visualizations to support exploration of such journals, as well as an examination of patient-provider collaboration in interpreting the journals. Drawing upon individual and collaborative interviews with patients and providers, we find that collaborative review helps improve data comprehension and build mutual trust. We also find a desire to use tools like our interactive visualizations within and beyond clinic appointments. We discuss these findings and present guidance for the design of future tools.

Many people struggle with efforts to make healthy behavior changes, such as healthy eating. Several existing approaches promote healthy eating, but present high barriers and yield limited engagement. As a lightweight alternative approach to promoting mindful eating, we introduce and examine crumbs: daily food challenges completed by consuming one food that meets the challenge. We examine crumbs through developing and deploying the iPhone application Food4Thought. In a 3 week field study with 61 participants, crumbs supported engagement and mindfulness while offering opportunities to learn about food. Our 2x2 study compared nutrition versus non-nutrition crumbs coupled with social versus non-social features. Nutrition crumbs often felt more purposeful to participants, but non-nutrition crumbs increased mindfulness more than nutrition crumbs. Social features helped sustain engagement and were important for engagement with non-nutrition crumbs. Social features also enabled learning about the variety of foods other people use to meet a challenge.

Recent research examines how and why people abandon self-tracking tools. We extend this work with new insights drawn from people reflecting on their experiences after they stop tracking, examining how designs continue to influence people even after abandonment. We further contrast prior work considering abandonment of health and wellness tracking tools with an exploration of why people abandon financial and location tracking tools, and we connect our findings to models of personal informatics. Surveying 193 people and interviewing 12 people, we identify six reasons why people stop tracking and five perspectives on life after tracking. We discuss these results and opportunities for design to consider life after self-tracking.

In situ self-report is widely used in human-computer interaction, ubiquitous computing, and for assessment and intervention in health and wellness. Unfortunately, it remains limited by high burdens. We examine unlock journaling as an alternative. Specifically, we build upon recent work to introduce single-slide unlock journaling gestures appropriate for health and wellness measures. We then present the first field study comparing unlock journaling with traditional diaries and notification-based reminders in self-report of health and wellness measures. We find unlock journaling is less intrusive than reminders, dramatically improves frequency of journaling, and can provide equal or better timeliness. Where appropriate to broader design needs, unlock journaling is thus an overall promising method for in situ self-report.

Patient-generated data is increasingly common in chronic disease care management. Smartphone applications and wearable sensors help patients more easily collect health information. However, current commercial tools often do not effectively support patients and providers in collaboration surrounding these data. This paper examines patient expectations and current collaboration practices around patient-generated data. We survey 211 patients, interview 18 patients, and re-analyze a dataset of 21 provider interviews. We find that collaboration occurs in every stage of self-tracking and that patients and providers create boundary negotiating artifacts to support the collaboration. Building upon current practices with patient-generated data, we use these theories of patient and provider collaboration to analyze misunderstandings and privacy concerns as well as identify opportunities to better support these collaborations. We reflect on the social nature of patient-provider collaboration to suggest future development of the stage-based model of personal informatics and the theory of boundary negotiating artifacts.

stage of their tool use. We explore how designs can support people when they lapse in tracking, considering how to design data representations for a person who lapses in Fitbit use. Through a survey of 141 people who had lapsed in using Fitbit, we identified three use patterns and four perspectives on tracking. Participants then viewed seven visual representations of their Fitbit data and seven approaches to framing this data. Participant Fitbit use and perspective on tracking influenced their preference, which we surface in a series of contrasts. Specifically, our findings guide selecting appropriate aggregations from Fitbit use (e.g., aggregate more when someone has less data), choosing an appropriate framing technique from tracking perspective (e.g., ensure framing aligns with how the person feels about tracking), and creating appropriate social comparisons (e.g., portray the person positively compared to peers). We conclude by discussing how these contrasts suggest new designs and opportunities in other tracking domains.

OBJECTIVE: To describe an interdisciplinary and methodological framework for applying single case study designs to self-experimentation in personalized health. The authors examine the framework's applicability to various health conditions and present an initial case study with irritable bowel syndrome (IBS). METHODS AND MATERIALS: An in-depth literature review was performed to develop the framework and to identify absolute and desired health condition requirements for the application of this framework. The authors developed mobile application prototypes, storyboards, and process flows of the framework using IBS as the case study. The authors conducted three focus groups and an online survey using a human-centered design approach for assessing the framework's feasibility. RESULTS: All 6 focus group participants had a positive view about our framework and volunteered to participate in future studies. Most stated they would trust the results because it was their own data being analyzed. They were most concerned about confounds, nonmeaningful measures, and erroneous assumptions on the timing of trigger effects. Survey respondents (N = 60) were more likely to be adherent to an 8- vs 12-day study length even if it meant lower confidence results. DISCUSSION: Implementation of the self-experimentation framework in a mobile application appears to be feasible for people with IBS. This framework can likely be applied to other health conditions. Considerations include the learning curve for teaching self-experimentation to non-experts and the challenges involved in operationalizing and customizing study designs. CONCLUSION: Using mobile technology to guide people through self-experimentation to investigate health questions is a feasible and promising approach to advancing personalized health.

Food choices are among the most frequent and important health decisions in everyday life, but remain notoriously difficult to capture. This work examines opportunities for lightweight photo-based capture in mobile food journals. We first report on a survey of 257 people, examining how they define healthy eating, their experiences and challenges with existing food journaling methods, and their ability to interpret nutritional information that can be captured in a food journal. We then report on interviews and a field study with 27 participants using a lightweight, photo-based food journal for between 4 to 8 weeks. We discuss mismatches between motivations and current designs, challenges of current approaches to food journaling, and opportunities for photos as an alternative to the pervasive but often inappropriate emphasis on quantitative tracking in mobile food journals.

Although food journaling is understood to be both important and difficult, little work has empirically documented the specific challenges people experience with food journals. We identify key challenges in a qualitative study combining a survey of 141 current and lapsed food journalers with analysis of 5,526 posts in community forums for three mobile food journals. Analyzing themes in this data, we find and discuss barriers to reliable food entry, negative nudges caused by current techniques, and challenges with social features. Our results motivate research exploring a wider range of approaches to food journal design and technology.

Current models of how people use personal informatics systems are largely based in behavior change goals. They do not adequately characterize the integration of self-tracking into everyday life by people with varying goals. We build upon prior work by embracing the perspective of lived informatics to propose a new model of personal informatics. We examine how lived informatics manifests in the habits of self-trackers across a variety of domains, first by surveying 105, 99, and 83 past and present trackers of physical activity, finances, and location and then by interviewing 22 trackers regarding their lived informatics experiences. We develop a model characterizing tracker processes of deciding to track and selecting a tool, elaborate on tool usage during collection, integration, and reflection as components of tracking and acting, and discuss the lapsing and potential resuming of tracking. We use our model to surface underexplored challenges in lived informatics, thus identifying future directions for personal informatics design and research.

We introduce the first forearm-based EMG input system that can recognize fine-grained thumb gestures, including left swipes, right swipes, taps, long presses, and more complex thumb motions. EMG signals for thumb motions sensed from the forearm are quite weak and require significant training data to classify. We therefore also introduce a novel approach for minimally-intrusive collection of labeled training data for always-available input devices. Our dual-observable input approach is based on the insight that interaction observed by multiple devices allows recognition by a primary device (e.g., phone recognition of a left swipe gesture) to create labeled training examples for another (e.g., forearm-based EMG data labeled as a left swipe). We implement a wearable prototype with dry EMG electrodes, train with labeled demonstrations from participants using their own phones, and show that our prototype can recognize common fine-grained thumb gestures and user-defined complex gestures.

As people continue to adopt technology based self tracking devices and applications, questions arise about how personal informatics tools can better support self tracker goals. This paper extends prior work on analyzing and summarizing self tracking data, with the goal of helping self trackers identify more meaningful and actionable findings. We begin by surveying physical activity self trackers to identify their goals and the factors they report influence their physical activity. We then define a cut as a subset of collected data with some shared feature, develop a set of cuts over location and physical activity data, and visualize those cuts using a variety of presentations. Finally, we conduct a month long field deployment with participants tracking their location and physical activity data and then using our methods to examine their data. We report on participant reactions to our methods and future design opportunities suggested by our work.

Interactive end-user training of machine learning systems has received significant attention as a tool for personalizing recognizers. However, most research limits end users to training a fixed set of application-defined concepts. This paper considers additional challenges that arise in end-user support for defining the number and nature of concepts that a system must learn to recognize. We develop BeatBox, a new system that enables end-user creation of custom beatbox recognizers and interactive adaptation of recognizers to an end user's technique, environment, and musical goals. BeatBox proposes rapid end-user exploration of variations in the number and nature of learned concepts, and provides end users with feedback on the reliability of recognizers learned for different potential combinations of percussive vocalizations. In a preliminary evaluation, we observed that end users were able to quickly create usable classifiers, that they explored different combinations of concepts to test alternative vocalizations and to refine classifiers for new musical contexts, and that learnability feedback was often helpful in alerting them to potential difficulties with a desired learning concept.

Pixel-based methods offer unique potential for modifying existing interfaces independent of their underlying implementation. Prior work has demonstrated a variety of modifications to existing interfaces, including accessibility enhancements, interface language translation, testing frameworks, and interaction techniques. But pixel-based methods have also been limited in their understanding of the interface and therefore the complexity of modifications they can support. This work examines deeper pixel-level understanding of widgets and the resulting capabilities of pixel-based runtime enhancements. Specifically, we present three new sets of methods: methods for pixel-based modeling of widgets in multiple states, methods for managing the combinatorial complexity that arises in creating a multitude of runtime enhancements, and methods for styling runtime enhancements to preserve consistency with the design of an existing interface. We validate our methods through an implementation of Moscovich et al.'s Sliding Widgets, a novel runtime enhancement that could not have been implemented with prior pixel-based methods.

Gesture-based interactions have become an essential part of the modern user interface. However, it remains challenging for developers to create gestures for their applications. This paper studies unistroke gestures, an important category of gestures defined by their single-stroke trajectories. We present Gesture Script, a tool for creating unistroke gesture recognizers. Gesture Script enhances example-based learning with interactive declarative guidance through rendering scripts and interactively trained parts. The structural information from the rendering scripts allows Gesture Script to synthesize gesture variations and generate a more accurate recognizer that also automatically extracts gesture attributes needed by applications. The results of our study with developers show that Gesture Script preserves the threshold of familiar example based gesture tools, while raising the ceiling of the recognizers created in such tools.

Pixel-based methods have the potential to fundamentally change how we build graphical interfaces, but remain difficult to implement. We introduce a new toolkit for pixel based enhancements, focused on two areas of support. Prefab Layers helps developers write interpretation logic that can be composed, reused, and shared to manage the multi-faceted nature of pixel-based interpretation. Prefab Annotations supports robustly annotating interface elements with metadata needed to enable runtime enhancements. Together, these help developers overcome subtle but critical dependencies between code and data. We validate our toolkit with (1) demonstrative applications and (2) a lab study that compares how developers build an enhancement using our toolkit versus state of the art methods. Our toolkit addresses core challenges faced by developers when building pixel based enhancements, potentially opening up pixel based systems to broader adoption.

Personal informatics applications in a variety of domains are increasingly enabled by low cost personal sensing. Although applications capture fine-grained activity for self reflection, sharing is generally limited to high level summaries. There are potential advantages to fine-grained sharing, but also potential harms. To help investigate this complex design space, we employ Value Sensitive Design to consider whether and how to share fine grained step activity. We identify key values and value tensions, and we develop scenarios to highlight these. We then design a set of data transformations that seek to maximize the benefits while minimizing the harms of detailed sharing. These include a novel approach to interactive modification of fine grained step data, allowing people to remove private data and using motif discovery to generate realistic replacement data. Finally, we conduct semi structured interviews with 12 participants examining these scenarios and transformations. We distill results into a set of design considerations for fine-grained physical activity sharing.

We present ReGroup, a novel end-user interactive machine learning system for helping people create custom, on demand groups in online social networks. As a person adds members to a group, ReGroup iteratively learns a probabilistic model of group membership specific to that group. ReGroup then uses its currently learned model to suggest additional members and group characteristics for filtering. Our evaluation shows that ReGroup is effective for helping people create large and varied groups, whereas traditional methods (searching by name or selecting from an alphabetical list) are better suited for small groups whose members can be easily recalled by name. By facilitating on demand group creation, ReGroup can enable in-context sharing and potentially encourage better online privacy practices. In addition, applying interactive machine learning to social network group creation introduces several challenges for designing effective end-user interaction with machine learning. We identify these challenges and discuss how we address them in ReGroup.

We present a general-purpose implementation of a target aware pointing technique, functional across an entire desktop and independent of application implementations. Specifically, we implement Grossman and Balakrishnan's Bubble Cursor, the fastest general pointing facilitation technique in the literature. Our implementation obtains the necessary knowledge of interface targets using a combination of pixel-level analysis and social annotation. We discuss the most novel aspects of our implementation, including methods for interactive creation and correction of pixel-level prototypes of interface elements and methods for interactive annotation of how the cursor should select identified elements. We also report on limitations of the Bubble Cursor unearthed by examining our implementation in the complexity of real-world interfaces. We therefore contribute important progress toward real-world deployment of an important family of techniques and shed light on the gap between understanding techniques in controlled settings versus behavior with real-world interfaces.

User interface toolkit research has traditionally assumed that developers have full control of an interface. This assumption is challenged by the mashup nature of many modern interfaces, in which different portions of a single interface are implemented by multiple, potentially mutually distrusting developers (e.g., an Android application embedding a third-party advertisement). We propose considering security as a primary goal for user interface toolkits. We motivate the need for security at this level by examining today's mashup scenarios, in which security and interface flexibility are not simultaneously achieved. We describe a security-aware user interface toolkit architecture that secures interface elements while providing developers with the flexibility and expressivity traditionally desired in a user interface toolkit. By challenging trust assumptions inherent in existing approaches, this architecture effectively addresses important interface-level security concerns.

End-user interactive machine learning is a promising tool for enhancing human productivity and capabilities with large unstructured data sets. Recent work has shown that we can create end-user interactive machine learning systems for specific applications. However, we still lack a generalized understanding of how to design effective end-user interaction with interactive machine learning systems. This work presents three explorations in designing for effective end-user interaction with machine learning in CueFlik, a system developed to support Web image search. These explorations demonstrate that interactions designed to balance the needs of end-users and machine learning algorithms can significantly improve the effectiveness of end-user interactive machine learning.

The rigidity and fragmentation of GUI toolkits are fundamentally limiting the progress and impact of interaction research. Pixel-based methods offer unique potential for addressing these challenges independent of the implementation of any particular interface or toolkit. This work builds upon Prefab, which enables the modification of existing interfaces. We present new methods for hierarchical models of complex widgets, real-time interpretation of interface content, and real-time interpretation of content and hierarchy throughout an entire interface. We validate our new methods through implementations of four applications: stencil-based tutorials, ephemeral adaptation, interface translation, and end-user interface customization. We demonstrate these enhancements in complex existing applications created from different user interface toolkits running on different operating systems.

As games grow in complexity, gameplay needs to provide players with powerful means of managing this complexity. One approach is to give automation tools to players. In this paper, we analyze an in-game automation tool, the Foldit cookbook, for the scientific discovery game Foldit. The cookbook allows players to write recipes that can automate their strategies. Through analysis of cookbook usage, we observe that players take advantage of social mechanisms in the game to share, run, and modify recipes. Further, players take advantage of both a simplified visual programming interface and a text-based scripting interface for creating recipes. This indicates that there is potential for using automation tools to disseminate expert knowledge, and that it is useful to provide support for multiple authoring styles, especially for games where the final game goal is unbounded or hard to attain.

Improvements in cameras, computer vision, and machine learning are enabling real-time object recognition in interactive systems. Reliable recognition of uninstrumented objects opens up exciting new scenarios using the real-world objects that surround us. At the same time, it introduces the need to understand and manage the uncertainty and ambiguities that are inherent to such sensing. This paper examines this problem in the context of LEGO OASIS, a camera and projector-based system that recognizes LEGO toys and augments them with projected digital content. We focus on an interaction language to model the creation and manipulation of relationships between physical objects and their digital capabilities. We use this set of abstractions to examine different notions of recognition errors and explore interactive approaches to overcoming fundamental challenges in interactive object-aware systems.

End-user interactive concept learning is a technique for interacting with large unstructured datasets, requiring insights from both human-computer interaction and machine learning. This note re-examines an assumption implicit in prior interactive machine learning research, that interaction should focus on the question "what class is this object?". We broaden interaction to include examination of multiple potential models while training a machine learning system. We evaluate this approach and find that people naturally adopt revision in the interactive machine learning process and that this improves the quality of their resulting models for difficult concepts.

Current chasms between applications implemented with different user interface toolkits make it difficult to implement and explore potentially important interaction techniques in new and existing applications, limiting the progress and impact of human-computer interaction research. We examine an approach based in the single most common characteristic of all graphical user interface toolkits, that they ultimately paint pixels to a display. We present Prefab, a system for implementing advanced behaviors through the reverse engineering of the pixels in graphical interfaces. Informed by how user interface toolkits paint interfaces, Prefab features a separation of the modeling of widget layout from the recognition of widget appearance. We validate Prefab in implementations of three applications: target-aware pointing techniques, Phosphor transitions, and Side Views parameter spectrums. Working only from pixels, we demonstrate a single implementation of these enhancements in complex existing applications created in different user interface toolkits running on different windowing systems.

Advances in cyber infrastructure for virtual observatories are poised to allow scientists from disparate fields to conduct experiments together, monitor large collections of instruments, and explore extensive archives of observed and simulated data. Such systems, however, focus on the `plumbing' and frequently ignore the critical importance of rich, 3D interactive visualization, asset management, and collaboration necessary for interdisciplinary communication. The NSF Ocean Observatories Initiative (OOI) is typical of modern observatory-oriented projects-its goal is to transform ocean science from an expeditionary science to an observatory science. This paper explores the design of an interactive tool to support this new way of conducting ocean science. Working directly with teams of scientists, we designed and deployed the Collaborative Ocean Visualization Environment (COVE). We then carried out three field evaluations of COVE: a multi-month deployment with the scientists and engineers of an observatory design team and two deployments at sea as the primary planning and collaboration platform on expeditionary cruises to map observatory sites and study geothermal vents.

We present Gestalt, a development environment designed to support the process of applying machine learning. While traditional programming environments focus on source code, we explicitly support both code and data. Gestalt allows developers to implement a classification pipeline, analyze data as it moves through that pipeline, and easily transition between implementation and analysis. An experiment shows this significantly improves the ability of developers to find and fix bugs in machine learning systems. Our discussion of Gestalt and our experimental observations provide new insight into general-purpose support for the machine learning process.

Graphical password systems have received significant attention as one potential solution to the need for more usable authentication, but nearly all prior work makes the unrealistic assumption of studying a single password. This paper presents the first study of multiple graphical passwords to systematically examine frequency of access to a graphical password, interference resulting from interleaving access to multiple graphical passwords, and patterns of access while training multiple graphical passwords. We find that all of these factors significantly impact the ease of authenticating using multiple facial graphical passwords. For example, participants who accessed four different graphical passwords per week were ten times more likely to completely fail to authenticate than participants who accessed a single password once per week. Our results underscore the need for more realistic evaluations of the use of multiple graphical passwords, have a number of implications for the adoption of graphical password systems, and provide a new basis for comparing proposed graphical password systems.

Although existing work has explored both information extraction and community content creation, most research has focused on them in isolation. In contrast, we see the greatest leverage in the synergistic pairing of these methods as two interlocking feedback cycles. This paper explores the potential synergy promised if these cycles can be made to accelerate each other by exploiting the same edits to advance both community content creation and learning-based information extraction. We examine our proposed synergy in the context of Wikipedia infoboxes and the Kylin information extraction system. After developing and refining a set of interfaces to present the verification of Kylin extractions as a non primary task in the context of Wikipedia articles, we develop an innovative use of Web search advertising services to study people engaged in some other primary task. We demonstrate our proposed synergy by analyzing our deployment from two complementary perspectives: (1) we show we accelerate community content creation by using Kylin's information extraction to significantly increase the likelihood that a person visiting a Wikipedia article as a part of some other primary task will spontaneously choose to help improve the article's infobox, and (2) we show we accelerate information extraction by using contributions collected from people interacting with our designs to significantly improve Kylin's extraction performance.

We present a novel method of dynamic C-D gain adaptation that improves target acquisition for users with motor impairments. Our method, called the Angle Mouse, adjusts the mouse C-D gain based on the deviation of angles sampled during movement. When angular deviation is low, the gain is kept high. When angular deviation is high, the gain is dropped, making the target bigger in motor-space. A key feature of the Angle Mouse is that, unlike most pointing facilitation techniques, it is target-agnostic, requiring no knowledge of target locations or dimensions. This means that the problem of distractor targets is avoided because adaptation is based solely on the user's behavior. In a study of 16 people, 8 of which had motor impairments, we found that the Angle Mouse improved motor-impaired pointing throughput by 10.3% over the Windows default mouse and 11.0% over sticky icons. For able-bodied users, there was no significant difference among the three techniques, as Angle Mouse throughput was within 1.2% of the default. Thus, the Angle Mouse improved pointing performance for users with motor impairments while remaining unobtrusive for able-bodied users.

How can we cull the facts we need from the overwhelming mass of information and misinformation that is the Web? The TextRunner extraction engine represents one approach, in which people pose keyword queries or simple questions and TextRunner returns concise answers based on tuples extracted from Web text. Unfortunately, the results returned by engines such as TextRunner include both informative facts (e.g., “the FDA banned ephedra”) and less useful statements (e.g., “the FDA banned products”). This paper therefore investigates filtering TextRunner results to enable people to better focus on interesting assertions. We first develop three distinct models of what assertions are likely to be interesting in response to a query. We then fully operationalize each of these models as a filter over TextRunner results. Finally, we develop a more sophisticated filter that combines the different models using relevance feedback. In a study of human ratings of the interestingness of TextRunner assertions, we show that our approach substantially enhances the quality of TextRunner results. Our best filter raises the fraction of interesting results in the top thirty from 41.6 % to 64.1%.

Recent work has examined infrastructure-mediated sensing as a practical, low-cost, and unobtrusive approach to sensing human activity in the physical world. This approach is based on the idea that human activities (e.g., running a dishwasher, turning on a reading light, or walking through a doorway) can be sensed by their manifestations in an environment's existing infrastructures (e.g., a home's water, electrical, and HVAC infrastructures). This paper presents HydroSense, a low-cost and easily-installed single-point sensor of pressure within a home's water infrastructure. HydroSense supports both identification of activity at individual water fixtures within a home (e.g., a particular toilet, a kitchen sink, a particular shower) as well as estimation of the amount of water being used at each fixture. We evaluate our approach using data collected in ten homes. Our algorithms successfully identify fixture events with 97.9% aggregate accuracy and can estimate water usage with error rates that are comparable to empirical studies of traditional utility-supplied water meters. Our results both validate our approach and provide a basis for future improvements.

Interaction with large unstructured datasets is difficult because existing approaches, such as keyword search, are not always suited to describing concepts corresponding to the distinctions people want to make within datasets. One possible solution is to allow end users to train machine learning systems to identify desired concepts, a strategy known as interactive concept learning. A fundamental challenge is to design systems that preserve end user flexibility and control while also guiding them to provide examples that allow the machine learning system to effectively learn the desired concept. This paper presents our design and evaluation of four new overview based approaches to guiding example selection. We situate our explorations within CueFlik, a system examining end user interactive concept learning in Web image search. Our evaluation shows our approaches not only guide end users to select better training examples than the best performing previous design for this application, but also reduce the impact of not knowing when to stop training the system. We discuss challenges for end user interactive concept learning systems and identify opportunities for future research on the effective design of such systems.

Web image search is difficult in part because a handful of keywords are generally insufficient for characterizing the visual properties of an image. Popular engines have begun to provide tags based on simple characteristics of images (such as tags for black and white images or images that contain a face), but such approaches are limited by the fact that it is unclear what tags end users want to be able to use in examining Web image search results. This paper presents CueFlik, a Web image search application that allows end users to quickly create their own rules for re ranking images based on their visual characteristics. End users can then re rank any future Web image search results according to their rule. In an experiment we present in this paper, end users quickly create effective rules for such concepts as "product photos", "portraits of people", and "clipart". When asked to conceive of and create their own rules, participants create such rules as "sports action shot" with images from queries for "basketball" and "football". CueFlik represents both a promising new approach to Web image search and an important study in end user interactive machine learning.

As statistical machine learning algorithms and techniques continue to mature, many researchers and developers see statistical machine learning not only as a topic of expert study, but also as a tool for software development. Extensive prior work has studied software development, but little prior work has studied software developers applying statistical machine learning. This paper presents interviews of eleven researchers experienced in applying statistical machine learning algorithms and techniques to human-computer interaction problems, as well as a study of ten participants working during a five-hour study to apply statistical machine learning algorithms and techniques to a realistic problem. We distill three related categories of difficulties that arise in applying statistical machine learning as a tool for software development: (1) difficulty pursuing statistical machine learning as an iterative and exploratory process, (2) difficulty understanding relationships between data and the behavior of statistical machine learning algorithms, and (3) difficulty evaluating the performance of statistical machine learning algorithms and techniques in the context of applications. This paper provides important new insight into these difficulties and the need for development tools that better support the application of statistical machine learning.

Controlling the privacy of online content is difficult and often confusing. We present a social access control where users devise simple questions testing shared knowledge instead of constructing authenticated accounts and explicit access control rules. We implemented a prototype and conducted studies to explore the context of photo sharing security, gauge the difficulty of creating shared knowledge questions, measure their resilience to adversarial attack, and evaluate user ability to understand and predict this resilience.

The global obesity epidemic has prompted our community to explore the potential for technology to play a stronger role in promoting healthier lifestyles. Although there are several examples of successful games based on focused physical interaction, persuasive applications that integrate into everyday life have had more mixed results. This underscores a need for designs that encourage physical activity while addressing fun, sustainability, and behavioral change. This note suggests a new perspective, inspired in part by the social nature of many everyday fitness applications and by the successful encouragement of long term play in massively multiplayer online games. We first examine the game design literature to distill a set of principles for discussing and comparing applications. We then use these principles to analyze an existing application. Finally, we present Kukini, a design for an everyday fitness game.

Many mobile machine learning applications require collecting and labeling data, and a traditional GUI on a mobile device may not be an appropriate or viable method for this task. This paper presents an alternative approach to mobile labeling of sensor data called VoiceLabel. VoiceLabel consists of two components: (1) a speech-based data collection tool for mobile devices, and (2) a desktop tool for offline segmentation of recorded data and recognition of spoken labels. The desktop tool automatically analyzes the audio stream to find and recognize spoken labels, and then presents a multimodal interface for reviewing and correcting data labels using a combination of the audio stream, the system's analysis of that audio, and the corresponding mobile sensor data. A study with ten participants showed that VoiceLabel is a viable method for labeling mobile sensor data. VoiceLabel also illustrates several key features that inform the design of other data labeling tools.

The Web is ephemeral. Pages change frequently, and it is nearly impossible to find data or follow a link after the underlying page evolves. We present Zoetrope, a system that enables interaction with the historicalWeb (pages, links, and embedded data) that would otherwise be lost to time. Using a number of novel interactions, the temporal Web can be manipulated, queried, and analyzed from the context of familar pages. Zoetrope is based on a set of operators for manipulating content streams. We describe these primitives and the associated indexing strategies for handling temporal Web data. They form the basis of Zoetrope and enable our construction of new temporal interactions and visualizations.

People have developed a variety of conventions for negotiating face to face interruptions. The physical distribution of teams, however, together with the use of computer mediated communication and awareness systems, fundamentally alters what information is available to a person considering an interruption of a remote collaborator. This paper presents a detailed comparison between self-reports of interruptibility, collected from participants over extended periods in their actual work environment, and estimates of this interruptibility, provided by a second set of participants based on audio and video recordings. Our results identify activities and environmental cues that affect participants' ability to correctly estimate interruptibility. We show, for example, that a closed office door had a significant effect on observers' estimation of interruptibility, but did not have an effect on participants' reports of their own interruptibility. We discuss our findings and their importance for successful design of computer-mediated communication and awareness systems.

Sensor based statistical models promise to support a variety of advances in human computer interaction, but building applications that use them is currently difficult and potential advances go unexplored. We present Subtle, a toolkit that removes some of the obstacles to developing and deploying applications using sensor based statistical models of human situations. Subtle provides an appropriate and extensible sensing library, continuous learning of personalized models, fully automated high level feature generation, and support for using learned models in deployed applications. By removing obstacles to developing and deploying sensor based statistical models, Subtle makes it easier to explore the design space surrounding sensor based statistical models of human situations. Subtle thus helps to move the focus of human computer interaction research onto applications and datasets, instead of the difficulties of developing and deploying sensor based statistical models.

In order to develop intelligent systems that attain the trust of their users, it is important to understand how users perceive such systems and develop those perceptions over time. We present an investigation into how users come to understand an intelligent system as they use it in their daily work. During a six-week field study, we interviewed eight office workers regarding the operation of a system that predicted their managers' interruptibility, comparing their mental models to the actual system model. Our results show that by the end of the study, participants were able to discount some of their initial misconceptions about what information the system used for reasoning about interruptibility. However, the overarching structures of their mental models stayed relatively stable over the course of the study. Lastly, we found that participants were able to give lay descriptions attributing simple machine learning concepts to the system despite their lack of technical knowledge. Our findings suggest an appropriate level of feedback for user interfaces of intelligent systems, provide a baseline level of complexity for user understanding, and highlight the challenges of making users aware of sensed inputs for such systems.

Programmers regularly use search as part of the development process, attempting to identify an appropriate API for a problem, seeking more information about an API, and seeking samples that show how to use an API. However, neither general-purpose search engines nor existing code search engines currently fit their needs, in large part because the information programmers need is distributed across many pages. We present Assieme, a Web search interface that effectively supports common programming search tasks by combining information from Web-accessible Java Archive (JAR) files, API documentation, and pages that include explanatory text and sample code. Assieme uses a novel approach to finding and resolving implicit references to Java packages, types, and members within sample code on the Web. In a study of programmers performing searches related to common programming tasks, we show that programmers obtain better solutions, using fewer queries, in the same amount of time spent using a general Web search interface.

The emergence of location-based computing promises new and compelling applications, but raises very real privacy risks. Existing approaches to privacy generally treat people as the entity of interest, often using a fidelity tradeoff to manage the costs and benefits of revealing a person's location. However, these approaches cannot be applied in some applications, as a reduction in precision can render location information useless. This is true of a category of applications that use location data collected from multiple people to infer such information as whether there is a traffic jam on a bridge, whether there are seats available in a nearby coffee shop, when the next bus will arrive, or if a particular conference room is currently empty. We present hitchhiking, a new approach that treats locations as the primary entity of interest. Hitchhiking removes the fidelity tradeoff by preserving the anonymity of reports without reducing the precision of location disclosures. We can therefore support the full functionality of an interesting class of location-based applications without introducing the privacy concerns that would otherwise arise.

The home deployment of sensor-based systems offers many opportunities, particularly in the area of using sensor-based systems to support aging in place by monitoring an elder's activities of daily living. But existing approaches to home activity recognition are typically expensive, difficult to install, or intrude into the living space. This paper considers the feasibility of a new approach that "reaches into the home" via the existing infrastructure. Specifically, we deploy a small number of low-cost sensors at critical locations in a home's water distribution infrastructure. Based on water usage patterns, we can then infer activities in the home. To examine the feasibility of this approach, we deployed real sensors into a real home for six weeks. Among other findings, we show that a model built on microphone-based sensors that are placed away from systematic noise sources can identify 100% of clothes washer usage, 95% of dishwasher usage, 94% of showers, 88% of toilet flushes, 73% of bathroom sink activity lasting ten seconds or longer, and 81% of kitchen sink activity lasting ten seconds or longer. While there are clear limits to what activities can be detected when analyzing water usage, our new approach represents a sweet spot in the tradeoff between what information is collected at what cost.

The computer and communication systems that office workers currently use tend to interrupt at inappropriate times or unduly demand attention because they have no way to determine when an interruption is appropriate. Sensor?based statistical models of human interruptibility offer a potential solution to this problem. Prior work to examine such models has primarily reported results related to social engagement, but it seems that task engagement is also important. Using an approach developed in our prior work on sensor?based statistical models of human interruptibility, we examine task engagement by studying programmers working on a realistic programming task. After examining many potential sensors, we implement a system to log low?level input events in a development environment. We then automatically extract features from these low?level event logs and build a statistical model of interruptibility. By correctly identifying situations in which programmers are non?interruptible and minimizing cases where the model incorrectly estimates that a programmer is non?interruptible, we can support a reduction in costly interruptions while still allowing systems to convey notifications in a timely manner.

A person seeking another person's attention is normally able to quickly assess how interruptible the other person currently is. Such assessments allow behavior that we consider natural, socially appropriate, or simply polite. This is in sharp contrast to current computer and communication systems, which are largely unaware of the social situations surrounding their usage and the impact that their actions have on these situations. If systems could model human interruptibility, they could use this information to negotiate interruptions at appropriate times, thus improving human computer interaction.This article presents a series of studies that quantitatively demonstrate that simple sensors can support the construction of models that estimate human interruptibility as well as people do. These models can be constructed without using complex sensors, such as vision-based techniques, and therefore their use in everyday office environments is both practical and affordable. Although currently based on a demographically limited sample, our results indicate a substantial opportunity for future research to validate these results over larger groups of office workers. Our results also motivate the development of systems that use these models to negotiate interruptions at socially appropriate times.

Current systems often create socially awkward interruptions or unduly demand attention because they have no way of knowing if a person is busy and should not be interrupted. Previous work has examined the feasibility of using sensors and statistical models to estimate human interruptibility in an office environment, but left open some questions about the robustness of such an approach. This paper examines several dimensions of robustness in sensor-based statistical models of human interruptibility. We show that real sensors can be constructed with sufficient accuracy to drive the predictive models. We also create statistical models for a much broader group of people than was studied in prior work. Finally, we examine the effects of training data quantity on the accuracy of these models and consider tradeoffs associated with different combinations of sensors. As a whole, our analyses demonstrate that sensor-based statistical models of human interruptibility can provide robust estimates for a variety of office workers in a range of circumstances, and can do so with accuracy as good as or better than people. Integrating these models into systems could support a variety of advances in human computer interaction and computer-mediated communication. Author Keywords Situationally appropriate interaction, managing human attention, sensor-based interfaces, context-aware computing, machine learning.

Recent work is beginning to reveal the potential of numerical optimization as an approach to generating interfaces and displays. Optimization-based approaches can often allow a mix of independent goals and constraints to be blended in ways that are difficult to describe algorithmically. While optimization-based techniques appear to offer several potential advantages, further research in this area is hampered by the lack of appropriate tools. Optimization toolkits do exist, but they typically require substantial specialized knowledge because they have been designed for traditional optimization problems.GADGET is an experimental toolkit to support optimization as an approach to interface and display generation. GADGET provides three core abstractions, initializers, iterations, and evaluations. An initializer creates an initial solution to be optimized, based on an existing algorithm or randomly. Iterations are responsible for transforming one potential solution into another, typically using methods that are at least partially random. Finally, evaluations are used for judging the different notions of goodness in a solution. Together with a evaluation standardization framework, support for generic properties integrated with an efficient lazy evaluation framework, and a library of reusable iterations and evaluations, the abstractions provided by GADGET simplify the development of optimization-based approaches to interface and display generation.

A person seeking someone else's attention is normally able to quickly assess how interruptible they are. This assessment allows for behavior we perceive as natural, socially appropriate, or simply polite. On the other hand, today's computer systems are almost entirely oblivious to the human world they operate in, and typically have no way to take into account the interruptibility of the user. This paper presents a Wizard of Oz study exploring whether, and how, robust sensor-based predictions of interruptibility might be constructed, which sensors might be most useful to such predictions, and how simple such sensors might be.The study simulates a range of possible sensors through human coding of audio and video recordings. Experience sampling is used to simultaneously collect randomly distributed self-reports of interruptibility. Based on these simulated sensors, we construct statistical models predicting human interruptibility and compare their predictions with the collected self-report data. The results of these models, although covering a demographically limited sample, are very promising, with the overall accuracy of several models reaching about 78%. Additionally, a model tuned to avoiding unwanted interruptions does so for 90% of its predictions, while retaining 75% overall accuracy.

Recent work is beginning to reveal the potential of numerical optimization as an approach to generating interfaces and displays. Optimization-based approaches can often allow a mix of independent goals and constraints to be blended in ways that would be difficult to describe algorithmically. While optimization-based techniques appear to offer several potential advantages, further research in this area is hampered by the lack of appropriate tools. This paper presents GADGET, an experimental toolkit to support optimization for interface and display generation. GADGET provides convenient abstractions of many optimization concepts. GADGET also provides mechanisms to help programmers quickly create optimizations, including an efficient lazy evaluation framework, a powerful and configurable optimization structure, and a library of reusable components. Together these facilities provide an appropriate tool to enable exploration of a new class of interface and display generation techniques.

In order to create and use rich custom appearances, designers are often forced to introduce an unnatural gap into the design process. For example, a designer creating a skin for a music player must separately specify the appearance of the elements in the music player skin and the mapping between these visual elements and the functionality provided by the music player. This gap between appearance and semantic meaning creates a number of problems. We present a set of techniques that allows designers to use their preferred drawing tool to specify both appearance and semantic meaning. We demonstrate our techniques in an unmodified version of Adobe Photoshop®, but our techniques are general and adaptable to nearly any layered drawing package.

Normally, the primary purpose of an information display is to convey information. If information displays can be aesthetically interesting, that might be an added bonus. This paper considers an experiment in reversing this imperative. It describes the Kandinsky system which is designed to create displays which are first aesthetically interesting, and then as an added bonus, able to convey information. The Kandinsky system works on the basis of aesthetic properties specified by an artist (in a visual form). It then explores a space of collages composed from information bearing images, using an optimization technique to find compositions which best maintain the properties of the artist's aesthetic expression.