OBJECTIVE: While non-invasive, cuffless blood pressure (BP) measurement has demonstrated relevancy in controlled environments, ambulatory measurement is important for hypertension diagnosis and control. We present both in-lab and ambulatory BP estimation results from a diverse cohort of participants. METHODS: Participants (N=1125, aged 21-85, 49.2% female, multiple hypertensive categories) had BP measured in-lab over a 24-hour period with a subset also receiving ambulatory measurements. Radial tonometry, photoplethysmography (PPG), electrocardiography (ECG), and accelerometry signals were collected simultaneously with auscultatory or oscillometric references for systolic (SBP) and diastolic blood pressure (DBP). Predictive models to estimate BP using a variety of sensor-based feature groups were evaluated against challenging baselines. RESULTS: Despite limited availability, tonometry-derived features showed superior performance compared to other feature groups and baselines, yieldingprediction errors of 0.32 ±9.8 mmHg SBP and 0.54 ±7.7 mmHg DBP in-lab, and 0.86 ±8.7 mmHg SBP and 0.75 ±5.9 mmHg DBP for 24-hour averages. SBP error standard deviation (SD) was reduced in normotensive (in-lab: 8.1 mmHg, 24-hr: 7.2 mmHg) and younger (in-lab: 7.8 mmHg, 24-hr: 6.7 mmHg) subpopulations. SBP SD was further reduced 15-20% when constrained to the calibration posture alone. CONCLUSION: Performance for normotensive and younger participants was superior to the general population across all feature groups. Reference type, posture relative to calibration, and controlled vs. ambulatory setting all impacted BP errors. SIGNIFICANCE: Results highlight the need for demographically diverse populations and challenging evaluation settings for BP estimation studies. We present the first public dataset of ambulatory tonometry and cuffless BP over a 24-hour period to aid in future cardiovascular research.

We present fingerIO, a novel fine-grained finger tracking solution for around-device interaction. FingerIO does not require instrumenting the finger with sensors and works even in the presence of occlusions between the finger and the device. We achieve this by transforming the device into an active sonar system that transmits inaudible sound signals and tracks the echoes of the finger at its microphones. To achieve sub-centimeter level tracking accuracies, we present an innovative approach that use a modulation technique commonly used in wireless communication called Orthogonal Frequency Division Multiplexing (OFDM). Our evaluation shows that fingerIO can achieve 2-D finger tracking with an average accuracy of 8 mm using the in-built microphones and speaker of a Samsung Galaxy S4. It also tracks subtle finger motion around the device, even when the phone is in the pocket. Finally, we prototype a smart watch form-factor fingerIO device and show that it can extend the interaction space to a 0.5×0.25 m2 region on either side of the device and work even when it is fully occluded from the finger.

Blood pressure (BP) is typically captured at irregular intervals, mostly in clinic environments. This approach treats BP as a static snapshot for health classification and largely ignores its value as a continuously fluctuat-ing measure. Recognizing that consumers are increasing-ly capturing health metrics through wearable devices, we explored BP measurement in relation to everyday living through a two-week field study with 34 adults. Based on questionnaires, measurement logs, and interviews, we examined participants' perceptions and attitudes to-wards BP variability and their associations of BP with aspects of their lives. We found that participants modi-fied their use of BP devices in response to BP variabil-ity, made associations with stress, food, and daily rou-tines, and revealed challenges with the design of current BP devices for personal use. We present design recom-mendations for BP use in everyday contexts and de-scribe strategies for re-framing BP capture and reporting.

This editorial introduction describes the aims and scope of the ACM Transactions on Interactive Intelligent Systems special issue on Activity Recognition for Interaction. It explains why activity recognition is becoming crucial as part of the cycle of interaction between users and computing systems, and it shows how the five articles selected for this special issue reflect this theme.

Physiological data in its different dimensions—bioelectrical, biomechanical, biochemical, or biophysical—and collected through existing sensors or specialized biomedical devices, image capture, or other sources is pushing the boundaries of physiological computing for human-computer interaction (HCI). Although physiological computing shows the potential to enhance the way in which people interact with digital content, systems remain challenging to design and build. The aim of this special issue is to present outstanding work related to use of physiological data in HCI, setting additional bases for next-generation computer interfaces and interaction experiences. Topics covered in this issue include methods and methodologies, human factors, the use of devices, and applications for supporting the development of emerging interfaces.

Though it has been asserted that "good design is honest", [42] deception exists throughout human-computer interaction research and practice. Because of the stigma associated with deception - in many cases rightfully so - the research community has focused its energy on eradicating malicious deception, and ignored instances in which deception is positively employed. In this paper we present the notion of benevolent deception, deception aimed at benefitting the user as well as the developer. We frame our discussion using a criminology-inspired model and ground components in various examples. We assert that this provides us with a set of tools and principles that not only helps us with system and interface design, but that opens new research areas. After all, as Cockton claims in his 2004 paper "Value-Centered HCI" [13], "Traditional disciplines have delivered truth. The goal of HCI is to deliver value."

Input modalities such as speech and gesture allow users to interact with computers without holding or touching a physical device, thus enabling at-a-distance interaction. It remains an open problem, however, to incorporate haptic feedback into such interaction. In this work, we explore the use of air vortex rings for this purpose. Unlike standard jets of air, which are turbulent and dissipate quickly, vortex rings can be focused to travel several meters and impart perceptible feedback. In this paper, we review vortex formation theory and explore specific design parameters that allow us to generate vortices capable of imparting haptic feedback. Applying this theory, we developed a prototype system called AirWave. We show through objective meas urements that AirWave can achieve spatial resolution of less than 10 cm at a distance of 2.5 meters. We further demonstrate through a user study that this can be used to direct tactile stimuli to different regions of the human body.

We harness the ability of people to perceive and interact with visual patterns in order to enhance the performance of a machine learning method. We show how we can collect evidence about how people optimize the parameters of an ensemble classification system using a tool that provides a visualization of misclassification costs. Then, we use these observations about human attempts to minimize cost in order to extend the performance of a state-of-the-art ensemble classification system. The study highlights opportunities for learning from evidence collected about human problem solving to refine and extend automated learning and inference.

Problem-solving procedures have been typically aimed at achieving well-defined goals or satisfying straightforward preferences. However, learners and solvers may often generate rich multiattribute results with procedures guided by sets of controls that define different dimensions of quality. We explore methods that enable people to explore and express preferences about the operation of classification models in supervised multiclass learning. We leverage a leave-one-out confusion matrix that provides users with views and real-time controls of a model space. The approach allows people to consider in an interactive manner the global implications of local changes in decision boundaries. We focus on kernel classifiers and show the effectiveness of the methodology on a variety of tasks.

Haptic devices that provide robust and realistic force feedback are generally grounded to counterweight the applied force, prohibiting their use in mobile devices. Many ungrounded force-feedback devices rely on the gyro effect to produce torques on the human body, but their active control systems render them extremely bulky for implementation in small mobile devices. We present GyroTab, a relatively flat handheld system that utilizes the gyro effect to provide torque feedback. GyroTab relies on the user to produce an input torque and provides feedback by opposing that torque, making its feedback reactive to the user's motion. We describe the implementation of GyroTab, discuss the kinds of feedback it generates, and explore some of the psychophysical results we obtained from a study with the device.

Computer vision and inertial measurement have made it possible for people to interact with computers using whole-body gestures. Although there has been rapid growth in the uses and applications of these systems, their ubiquity has been limited by the high cost of heavily instrumenting either the environment or the user. In this paper, we use the human body as an antenna for sensing whole-body gestures. Such an approach requires no instrumentation to the environment, and only minimal instrumentation to the user, and thus enables truly mobile applications. We show robust gesture recognition with an average accuracy of 93% across 12 whole-body gestures, and promising results for robust location classification within a building. In addition, we demonstrate a real-time interactive system which allows a user to interact with a computer using whole-body gestures

Gesture is becoming an increasingly popular means of interacting with computers. However, it is still relatively costly to deploy robust gesture recognition sensors in existing mobile platforms. We present SoundWave, a technique that leverages the speaker and microphone already embedded in most commodity devices to sense in-air gestures around the device. To do this, we generate an inaudible tone, which gets frequency-shifted when it reflects off moving objects like the hand. We measure this shift with the microphone to infer various gestures. In this note, we describe the phenomena and detection algorithm, demonstrate a variety of gestures, and present an informal evaluation on the robustness of this approach across different devices and people.

Researchers have explored a variety of technologies that enable a single display to simultaneously present different content when viewed from different angles or by different people. These displays provide new functionalities such as personalized views for multiple users, privacy protection, and stereoscopic 3D displays. However, current multi-view displays rely on special hardware, thus significantly limiting their availability to consumers and adoption in everyday scenarios. In this paper, we present a pure software solution (i.e. with no hardware modification) that allows us to present two independent views concurrently on the most widely used and affordable type of LCD screen, namely Twisted Nematic (TN). We achieve this by exploiting a technical limitation of the technology which causes these LCDs to show varying brightness and color depending on the viewing angle. We describe our technical solution as well as demonstrate example applications in everyday scenarios.

The awareness that hospital patients have of the people and events surrounding their care has a dramatic impact on satisfaction and clinical outcomes. However, patients are often under-informed about even basic aspects of their care. In this work, we hypothesize that mobile devices - which are increasingly available to patients - can be used as real-time information conduits to improve patient awareness and consequently improve patient care. To better understand the unique affordances that mobile devices offer in the hospital setting, we provided twenty-five patients with mobile phones that presented a dynamic, interactive report on their progress, care plan, and care team throughout their emergency department stay. Through interviews with these patients, their visitors, and hospital staff, we explore the benefits and challenges of using the mobile phone as an information display, finding overall that this is a promising approach to improving patient awareness. Furthermore, we demonstrate that only a small number of technology challenges remain before such a system could be deployed without researcher intervention.

Wearable sensor systems have been used in the ubiquitous computing community and elsewhere for applications such as activity and gesture recognition, health and wellness monitoring, and elder care. Although the power consumption of accelerometers has already been highly optimized, this work introduces a novel sensing approach which lowers the power requirement for motion sensing by orders of magnitude. We present an ultra-low-power method for passively sensing body motion using static electric fields by measuring the voltage at any single location on the body. We present the feasibility of using this sensing approach to infer the amount and type of body motion anywhere on the body and demonstrate an ultra-low-power motion detector used to wake up more power-hungry sensors. The sensing hardware consumes only 3.3 μW, and wake-up detection is done using an additional 3.3 μW (6.6 μW total).

Despite the pervasiveness of markets in our lives, little is known about the role of user interfaces (UIs) in promoting good decisions in market domains. How does the way we display market information to end users, and the set of choices we offer, influence users' decisions? In this paper, we introduce a new research agenda on "market user interface design." Our goal is to find the optimal market UI, taking into account that users incur cognitive costs and are boundedly rational. Via lab experiments we systematically explore the market UI design space, and we study the automatic optimization of market UIs given a behavioral (quantal response) model of user behavior. Surprisingly, we find that the behaviorally-optimized UI performs worse than the standard UI, suggesting that the quantal response model did not predict user behavior well. Subsequently, we identify important behavioral factors that are missing from the user model, including loss aversion and position effects, which motivates follow-up studies. Furthermore, we find significant differences between individual users in terms of rationality. This suggests future research on personalized UI designs, with interfaces that are tailored towards each individual user's needs, capabilities, and preferences.

We exploit the falloff of acuity in the visual periphery to accelerate graphics computation by a factor of 5-6 on a desktop HD display (1920x1080). Our method tracks the user's gaze point and renders three image layers around it at progressively higher angular size but lower sampling rate. The three layers are then magnified to display resolution and smoothly composited. We develop a general and efficient antialiasing algorithm easily retrofitted into existing graphics code to minimize "twinkling" artifacts in the lower-resolution layers. A standard psychophysical model for acuity falloff assumes that minimum detectable angular size increases linearly as a function of eccentricity. Given the slope characterizing this falloff, we automatically compute layer sizes and sampling rates. The result looks like a full-resolution image but reduces the number of pixels shaded by a factor of 10-15. We performed a user study to validate these results. It identifies two levels of foveation quality: a more conservative one in which users reported foveated rendering quality as equivalent to or better than non-foveated when directly shown both, and a more aggressive one in which users were unable to correctly label as increasing or decreasing a short quality progression relative to a high-quality foveated reference. Based on this user study, we obtain a slope value for the model of 1.32-1.65 arc minutes per degree of eccentricity. This allows us to predict two future advantages of foveated rendering: (1) bigger savings with larger, sharper displays than exist currently (e.g. 100 times speedup at a field of view of 70° and resolution matching foveal acuity), and (2) a roughly linear (rather than quadratic or worse) increase in rendering cost with increasing display field of view, for planar displays at a constant sharpness.

End-user interactive machine learning is a promising tool for enhancing human productivity and capabilities with large unstructured data sets. Recent work has shown that we can create end-user interactive machine learning systems for specific applications. However, we still lack a generalized understanding of how to design effective end-user interaction with interactive machine learning systems. This work presents three explorations in designing for effective end-user interaction with machine learning in CueFlik, a system developed to support Web image search. These explorations demonstrate that interactions designed to balance the needs of end-users and machine learning algorithms can significantly improve the effectiveness of end-user interactive machine learning.

Touch sensing and computer vision have made human-computer interaction possible in environments where keyboards, mice, or other handheld implements are not available or desirable. However, the high cost of instrumenting environments limits the ubiquity of these technologies, particularly in home scenarios where cost constraints dominate installation decisions. Fortunately, home environments frequently offer a signal that is unique to locations and objects within the home: electromagnetic noise. In this work, we use the body as a receiving antenna and leverage this noise for gestural interaction. We demonstrate that it is possible to robustly recognize touched locations on an uninstrumented home wall using no specialized sensors. We conduct a series of experiments to explore the capabilities that this new sensing modality may offer. Specifically, we show robust classification of gestures such as the position of discrete touches around light switches, the particular light switch being touched, which appliances are touched, differentiation between hands, as well as continuous proximity of hand to the switch, among others. We close by discussing opportunities, limitations, and future work.

Patients' basic understanding of clinical events has been shown to dramatically improve patient care. We propose that the automatic generation of very short micro-explanations, suitable for real-time delivery in clinical settings, can transform patient care by giving patients greater awareness of key events in their electronic medical record. We present results of a survey study indicating that it may be possible to automatically generate such explanations by extracting individual sentences from consumer-facing Web pages. We further inform future work by characterizing physician and non-physician responses to a variety of Web-extracted explanations of medical lab tests.

End-user interactive concept learning is a technique for interacting with large unstructured datasets, requiring insights from both human-computer interaction and machine learning. This note re-examines an assumption implicit in prior interactive machine learning research, that interaction should focus on the question "what class is this object?". We broaden interaction to include examination of multiple potential models while training a machine learning system. We evaluate this approach and find that people naturally adopt revision in the interactive machine learning process and that this improves the quality of their resulting models for difficult concepts.

We present Skinput, a technology that appropriates the human body for acoustic transmission, allowing the skin to be used as an input surface. In particular, we resolve the location of finger taps on the arm and hand by analyzing mechanical vibrations that propagate through the body. We collect these signals using a novel array of sensors worn as an armband. This approach provides an always available, naturally portable, and on-body finger input system. We assess the capabilities, accuracy and limitations of our technique through a two-part, twenty-participant user study. To further illustrate the utility of our approach, we conclude with several proof-of-concept applications we developed.

Interest has been growing within HCI on the use of machine learning and reasoning in applications to classify such hidden states as user intentions, based on observations. HCI researchers with these interests typically have little expertise in machine learning and often employ toolkits as relatively fixed "black boxes" for generating statistical classifiers. However, attempts to tailor the performance of classifiers to specific application requirements may require a more sophisticated understanding and custom-tailoring of methods. We present ManiMatrix, a system that provides controls and visualizations that enable system builders to refine the behavior of classification systems in an intuitive manner. With ManiMatrix, users directly refine parameters of a confusion matrix via an interactive cycle of re-classification and visualization. We present the core methods and evaluate the effectiveness of the approach in a user study. Results show that users are able to quickly and effectively modify decision boundaries of classifiers to tai-lor the behavior of classifiers to problems at hand.

Recent work in muscle sensing has demonstrated the poten-tial of human-computer interfaces based on finger gestures sensed from electrodes on the upper forearm. While this approach holds much potential, previous work has given little attention to sensing finger gestures in the context of three important real-world requirements: sensing hardware suitable for mobile and off-desktop environments, elec-trodes that can be put on quickly without adhesives or gel, and gesture recognition techniques that require no new training or calibration after re-donning a muscle-sensing armband. In this note, we describe our approach to over-coming these challenges, and we demonstrate average clas-sification accuracies as high as 86 % for pinching with one of three fingers in a two-session, eight-person experiment.

The Internet has allowed market-based systems to become increasingly pervasive. In this paper we explore the role of user interface (UI) design for these markets. Different UIs induce different mental models which in turn determine how users understand and interact with a market. Thus, the intersection of UI design and economics is a novel and important research area. We make three contributions at this intersection. First, we present a novel design paradigm which we call hidden markets. The primary goal of hidden markets is to hide as much of the market complexities as possible. Second, we explore this new design paradigm using one particular example: a P2P backup application. We explain the market underlying this system and provide a detailed description of the new UI we developed. Third, we present results from a formative usability study. Our findings indicate that a number of users could benefit from a market-based P2P backup system. Most users intuitively understood the give & take principle as well as the bundle constraints of the market. However, the pricing aspect was difficult to discover/understand for many users and thus needs further investigation. Overall, the results are encouraging and show promise for the hidden market paradigm.

Electronic medical records are increasingly comprehensive, and this vast repository of information has already contributed to medical efficiency and hospital procedure. However, this information is not typically accessible to patients, who are frequently under-informed and unclear about their own hospital courses. In this paper, we propose a design for in-room, patient-centric information displays, based on iterative design with physicians. We use this as the basis for a Wizard-of-Oz study in an emergency department, to assess patient and provider responses to in-room information displays. 18 patients were presented with real-time information displays based on their medical records. Semi-structured interviews with patients, family members, and hospital staff reveal that subjective response to in-room displays was overwhelmingly positive, and through these interviews we elicited guidelines regarding specific information types, privacy, use cases, and information presentation techniques. We describe these findings, and we discuss the feasibility of a fully-automatic implementation of our design.

When information is known only to friends in a social network, traditional crowdsourcing mechanisms struggle to motivate a large enough user population and to ensure accuracy of the collected information. We thus introduce friendsourcing, a form of crowdsourcing aimed at collecting accurate information available only to a small, socially-connected group of individuals. Our approach to friendsourcing is to design socially enjoyable interactions that produce the desired information as a side effect. We focus our analysis around Collabio, a novel social tagging game that we developed to encourage friends to tag one another within an online social network. Collabio encourages friends, family, and colleagues to generate useful information about each other. We describe the design space of incentives in social tagging games and evaluate our choices by a combination of usage log analysis and survey data. Data acquired via Collabio is typically accurate and augments tags that could have been found on Facebook or the Web. To complete the arc from data collection to application, we produce a trio of prototype applications to demonstrate how Collabio tags could be utilized: an aggregate tag cloud visualization, a personalized RSS feed, and a question and answer system. The social data powering these applications enables them to address needs previously difficult to support, such as question answering for topics comprehensible only to a few of a user's friends.

Previous research has shown that faceted browsing is effective and enjoyable in searching and browsing large collections of data. In this work, we explore the efficacy of interactive visualization systems in supporting exploration and sensemaking within faceted datasets. To do this, we developed an interactive visualization system called FacetLens, which exposes trends and relationships within faceted datasets. FacetLens implements linear facets to enable users not only to identify trends but also to easily compare several trends simultaneously. Furthermore, it offers pivot operations to allow users to navigate the faceted dataset using relationships between items. We evaluate the utility of the system through a description of insights gained while experts used the system to explore the CHI publication repository as well as a database of funding grant data, and report a formative user study that identified usability issues.

Machine learning is an increasingly used computational tool within human-computer interaction research. While most researchers currently utilize an iterative approach to refining classifier models and performance, we propose that ensemble classification techniques may be a viable and even preferable alternative. In ensemble learning, algorithms combine multiple classifiers to build one that is superior to its components. In this paper, we present EnsembleMatrix, an interactive visualization system that presents a graphical view of confusion matrices to help users understand relative merits of various classifiers. EnsembleMatrix allows users to directly interact with the visualizations in order to explore and build combination models. We evaluate the efficacy of the system and the approach in a user study. Results show that users are able to quickly combine multiple classifiers operating on multiple feature sets to produce an ensemble classifier with accuracy that approaches best-reported performance classifying images in the CalTech-101 dataset.

Previous research has explored how technology can motivate healthy living in social groups such as friends and coworkers. However, little research has focused on the implications of collecting, sharing, and reflecting upon health information within families. To explore this domain, we conducted a study that consisted of a week-long journaling activity followed by semi-structured interviews and formative design activities with 15 families (66 people). We identified four areas in which these practices are unique in a family context. Based on these findings we propose preliminary considerations for technologies that effectively support family reflections on health data.

Interaction with large unstructured datasets is difficult because existing approaches, such as keyword search, are not always suited to describing concepts corresponding to the distinctions people want to make within datasets. One possible solution is to allow end users to train machine learning systems to identify desired concepts, a strategy known as interactive concept learning. A fundamental challenge is to design systems that preserve end user flexibility and control while also guiding them to provide examples that allow the machine learning system to effectively learn the desired concept. This paper presents our design and evaluation of four new overview based approaches to guiding example selection. We situate our explorations within CueFlik, a system examining end user interactive concept learning in Web image search. Our evaluation shows our approaches not only guide end users to select better training examples than the best performing previous design for this application, but also reduce the impact of not knowing when to stop training the system. We discuss challenges for end user interactive concept learning systems and identify opportunities for future research on the effective design of such systems.

We present Collabio, a social tagging game within an online social network that encourages friends to tag one another. Collabio's approach of incentivizing members of the social network to generate information about each other produces personalizing information about its users. We report usage log analysis, survey data, and a rating exercise demonstrating that Collabio tags are accurate and augment information that could have been scraped online.

Many patients with paralyzing injuries or medical conditions retain the use of their cranial nerves, which control the eyes, jaw, and tongue. While researchers have explored eye-tracking and speech technologies for these patients, we believe there is potential for directly sensing explicit tongue movement for controlling computers. In this paper, we describe a novel approach of using infrared optical sensors embedded within a dental retainer to sense tongue gestures. We describe an experiment showing our system effectively discriminating between four simple gestures with over 90% accuracy. In this experiment, users were also able to play the popular game Tetris with their tongues. Finally, we present lessons learned and opportunities for future work.

Previous work has demonstrated the viability of applying offline analysis to interpret forearm electromyography (EMG) and classify finger gestures on a physical surface. We extend those results to bring us closer to using muscle-computer interfaces for always-available input in real-world applications. We leverage existing taxonomies of natural human grips to develop a gesture set covering interaction in free space even when hands are busy with other objects. We present a system that classifies these gestures in real-time and we introduce a bi-manual paradigm that enables use in interactive systems. We report experimental results demonstrating four-finger classification accuracies averaging 79% for pinching, 85% while holding a travel mug, and 88% when carrying a weighted bag. We further show generalizability across different arm postures and explore the tradeoffs of providing real-time visual feedback.

We present a new interaction framework for collaborating in multiple display environments (MDEs) and report results from a field study investigating its use in an authentic work setting. Our interaction framework, IMPROMPTU, allows users to share task information across displays via off-the-shelf applications, to jointly interact with information for focused problem solving and to place information on shared displays for discussion and reflection. Our framework also includes a lightweight interface for performing these and related actions. A three week field study of our framework was conducted in the domain of face-to-face group software development. Results show that teams utilized almost every feature of the framework in support of a wide range of development-related activities. The framework was used most to facilitate opportunistic collaboration involving task information. Teams reported wanting to continue using the framework as they found value in it overall.

Web image search is difficult in part because a handful of keywords are generally insufficient for characterizing the visual properties of an image. Popular engines have begun to provide tags based on simple characteristics of images (such as tags for black and white images or images that contain a face), but such approaches are limited by the fact that it is unclear what tags end users want to be able to use in examining Web image search results. This paper presents CueFlik, a Web image search application that allows end users to quickly create their own rules for re ranking images based on their visual characteristics. End users can then re rank any future Web image search results according to their rule. In an experiment we present in this paper, end users quickly create effective rules for such concepts as "product photos", "portraits of people", and "clipart". When asked to conceive of and create their own rules, participants create such rules as "sports action shot" with images from queries for "basketball" and "football". CueFlik represents both a promising new approach to Web image search and an important study in end user interactive machine learning.

While proponents of adaptive user interfaces tout potential performance gains, critics argue that adaptation's unpredictability may disorient users, causing more harm than good. We present a study that examines the relative effects of predictability and accuracy on the usability of adaptive UIs. Our results show that increasing predictability and accuracy led to strongly improved satisfaction. Increasing accuracy also resulted in improved performance and higher utilization of the adaptive interface. Contrary to our expectations, improvement in accuracy had a stronger effect on performance, utilization and some satisfaction ratings than the improvement in predictability.

A reliable and unobtrusive measurement of working memory load could be used to evaluate the efficacy of interfaces and to provide real-time user-state information to adaptive systems. In this paper, we describe an experiment we con-ducted to explore some of the issues around using an elec-troencephalograph (EEG) for classifying working memory load. Within this experiment, we present our classification methodology, including a novel feature selection scheme that seems to alleviate the need for complex drift modeling and artifact rejection. We demonstrate classification accuracies of up to 99% for 2 memory load levels and up to 88% for 4 levels. We also present results suggesting that we can do this with shorter windows, much less training data, and a smaller number of EEG channels, than reported previously. Finally, we show results suggesting that the models we construct transfer across variants of the task, implying some level of generality. We believe these findings extend prior work and bring us a step closer to the use of such technologies in HCI research.

We explore the feasibility of muscle-computer interfaces (muCIs): an interaction methodology that directly senses and decodes human muscular activity rather than relying on physical device actuation or user actions that are externally visible or audible. As a first step towards realizing the mu-CI concept, we conducted an experiment to explore the potential of exploiting muscular sensing and processing technologies for muCIs. We present results demonstrating accurate gesture classification with an off-the-shelf electromyography (EMG) device. Specifically, using 10 sensors worn in a narrow band around the upper forearm, we were able to differentiate position and pressure of finger presses, as well as classify tapping and lifting gestures across all five fingers. We conclude with discussion of the implications of our results for future muCI designs.

In this paper, we present Human-Aided Computing, an approach that uses an electroencephalograph (EEG) device to measure the presence and outcomes of implicit cognitive processing, processing that users perform automatically and may not even be aware of. We describe a classification system and present results from two experiments as proof-of-concept. Results from the first experiment showed that our system could classify whether a user was looking at an image of a face or not, even when the user was not explicitly trying to make this determination. Results from the second experiment extended this to animals and inanimate object categories as well, suggesting generality beyond face recognition. We further show that we can improve classification accuracies if we show images multiple times, potentially to multiple people, attaining well above 90% classification accuracies with even just ten presentations.

Human-aided computing proposes using information measured directly from the human brain in order to perform useful tasks. In this paper, we extend this idea by fusing computer vision-based processing and processing done by the human brain in order to build more effective object categorization systems. Specifically, we use an electroencephalograph (EEG) device to measure the subconscious cognitive processing that occurs in the brain as users see images, even when they are not trying to explicitly classify them. We present a novel framework that combines a discriminative visual category recognition system based on the Pyramid Match Kernel (PMK) with information derived from EEG measurements as users view images. We propose a fast convex kernel alignment algorithm to effectively combine the two sources of information. Our approach is validated with experiments using real-world data, where we show significant gains in classification accuracy. We analyze the properties of this information fusion method by examining the relative contributions of the two modalities, the errors arising from each source, and the stability of the combination in repeated experiments.

We explore the opportunity to harness electroencephalograph (EEG) signals generated during human visual processing to enhance computer vision systems. We review the challenging task of categorizing objects, such as faces, in images and then describe methods that can be used to combine the complementary competencies of human and machine computation to achieve improved recognition performance. We present the results of several experiments where brain signals, recorded from people examining images, are used to enhance the performance of vision systems on categorization tasks. We find that significant gains in classification accuracy can be achieved with the human-aided vision systems.

Software can now track which computer applications and documents you use. This provides us with the potential to help end-users recall past activities for tasks such as status reporting. We describe findings from field observations of eight participants writing their status reports. We observed interesting trends, including the reliance on memory triggers, which were either retrieved from explicit self-reminders, from implicit breadcrumbs left while performing their tasks or directly from memory. Participants perceived spending relatively short amounts of time composing their status reports, suggesting that any technology solution must offer dramatic improvements over current practice.

Using a notebook to sketch designs, reflect on a topic, or capture and extend creative ideas are examples of active note taking tasks. Optimal experience for such tasks demands concentration without interruption. Yet active note taking may also require reference documents or emails from team members. InkSeine is a Tablet PC application that supports active note taking by coupling a pen-and-ink interface with an in situ search facility that flows directly from a user's ink notes (Fig. 1). InkSeine integrates four key concepts: it leverages preexisting ink to initiate a search; it provides tight coupling of search queries with application content; it persists search queries as first class objects that can be commingled with ink notes; and it enables a quick and flexible workflow where the user may freely interleave inking, searching, and gathering content. InkSeine offers these capabilities in an interface that is tailored to the unique demands of pen input, and that maintains the primacy of inking above all other tasks.

Online pick'em games, such as the recent NCAA college basketball March Madness tournament, form a large and rapidly growing industry. In these games, players make predictions on a tournament bracket that defines which competitors play each other and how they proceed toward a single champion. Throughout the course of the tournament, players monitor the brackets to track progress and to compare predictions made by multiple players. This is often a complex sensemaking task. The classic bracket visualization was designed for use on paper and utilizes an incrementally additive system in which the winner of each match-up is rewritten in the next round as the tournament progresses. Unfortunately, this representation requires a significant amount of space and makes it relatively difficult to get a quick overview of the tournament state since competitors take arbitrary paths through the static bracket. In this paper, we present AdaptiviTree, a novel visualization that adaptively deforms the representation of the tree and uses its shape to convey outcome information. AdaptiviTree not only provides a more compact and understandable representation, but also allows overlays that display predictions as well as other statistics. We describe results from a lab study we conducted to explore the efficacy of AdaptiviTree, as well as from a deployment of the system in a recent real-world sports tournament.

For decades, researchers have presented different adaptive user interfaces and discussed the pros and cons of adaptation on task performance and satisfaction. Little research, however, has been directed at isolating and understanding those aspects of adaptive interfaces which make some of them successful and others not. We have designed and implemented three adaptive graphical interfaces and evaluated them in two experiments along with a non-adaptive baseline. In this paper we synthesize our results with previous work and discuss how different design choices and interactions affect the success of adaptive graphical user interfaces.

Accessing and manipulating occluded content in layered 2D drawings can be difficult. This paper characterizes a design space of techniques that facilitate access to occluded content. In addition, we introduce two new tools, Tumbler and Splatter, which represent unexplored areas of the design space. Finally, we present results of a study that contrasts these two tools against the traditional scene index used in most drawing applications. Results show that Splatter is comparable to and can be better than the scene index. Our findings allow us to understand the inherent design tradeoffs, and to identify areas for further improvement.

Information workers often have to balance many tasks and interruptions. In this work, we explore peripheral display techniques that improve multitasking efficiency by helping users maintain task flow, know when to resume tasks, and more easily reacquire tasks. Specifically, we compare two types of abstraction that provide different task information: semantic content extraction, which displays only the most relevant content in a window, and change detection, which signals when a change has occurred in a window (all de-signed as modifications to Scalable Fabric [17]). Results from our user study suggest that semantic content extraction improves multitasking performance more so than either change detection or our base case of scaling. Results also show that semantic content extraction provides significant benefits to task flow, resumption timing, and reacquisition. We discuss the implication of these findings on the design of peripheral interfaces that support multitasking.

Sometimes users fail to notice a change that just took place on their display. For example, the user may have accidentally deleted an icon or a remote collaborator may have changed settings in a control panel. Animated transitions can help, but they force users to wait for the animation to complete. This can be cumbersome, especially in situations where users did not need an explanation. We propose a different approach. Phosphor objects show the outcome of their transition instantly; at the same time they explain their change in retrospect. Manipulating a phosphor slider, for example, leaves an afterglow that illustrates how the knob moved. The parallelism of instant outcome and explanation supports both types of users. Users who already understood the transition can continue interacting without delay, while those who are inexperienced or may have been distracted can take time to view the effects at their own pace. We present a framework of transition designs for widgets, icons, and objects in drawing programs. We evaluate phosphor objects in two user studies and report significant performance benefits for phosphor objects.

Modern brain sensing technologies provide a variety of methods for detecting specific forms of brain activity. In this paper, we present an initial step in exploring how these technologies may be used to perform task classification and applied in a relevant manner to HCI research. We describe two experiments showing successful classification between tasks using a low-cost off-the-shelf electroencephalograph (EEG) system. In the first study, we achieved a mean classification accuracy of 84.0% in subjects performing one of three cognitive tasks - rest, mental arithmetic, and mental rotation - while sitting in a controlled posture. In the second study, conducted in more ecologically valid setting for HCI research, we attained a mean classification accuracy of 92.4% using three tasks that included non-cognitive features: a relaxation task, playing a PC based game without opponents, and engaging opponents within the game. Throughout the paper, we provide lessons learned and discuss how HCI researchers may utilize these technologies in their work.

With advances in pen-based computing devices, handwriting has become an increasingly popular input modality. Researchers have put considerable effort into building intelligent recognition systems that can translate handwriting to text with increasing accuracy. However, handwritten input is inherently ambiguous, and these systems will always make errors. Unfortunately, work on error recovery mechanisms has mainly focused on interface innovations that allow users to manually transform the erroneous recognition result into the intended one. In our work, we propose a mixed-initiative approach to error correction. We describe CueTIP, a novel correction interface that takes advantage of the recognizer to continually evolve its results using the additional information from user corrections. This significantly reduces the number of actions required to reach the intended result. We present a user study showing that CueTIP is more efficient and better preferred for correcting handwriting recognition errors. Grounded in the discussion of CueTIP, we also present design principles that may be applied to mixed-initiative correction interfaces in other domains.

Large wall-sized displays are becoming prevalent. Although researchers have articulated qualitative benefits of group work on large displays, little work has been done to quantify the benefits for individual users. In this article we present four experiments comparing the performance of users working on a large projected wall display to that of users working on a standard desktop monitor. In these experiments, we held the visual angle constant by adjusting the viewing distance to each of the displays. Results from the first two experiments suggest that physically large displays, even when viewed at identical visual angles as smaller ones, help users perform better on mental rotation tasks. We show through the experiments how these results may be attributed, at least in part, to large displays immersing users within the problem space and biasing them into using more efficient cognitive strategies. In the latter two experiments, we extend these results, showing the presence of these effects with more complex tasks, such as 3D navigation and mental map formation and memory. Results further show that the effects of physical display size are independent of other factors that may induce immersion, such as interactivity and mental aids within the virtual environments. We conclude with a general discussion of the findings and possibilities for future work.

The dominant paradigm for searching and browsing large data stores is text-based: presenting a scrollable list of search results in response to textual search term input. While this works well for the Web, there is opportunity for improvement in the domain of personal information stores, which tend to have more heterogeneous data and richer metadata. In this paper, we introduce FacetMap, an interactive, query-driven visualization, generalizable to a wide range of metadata-rich data stores. FacetMap uses a visual metaphor for both input (selection of metadata facets as filters) and output. Results of a user study provide insight into tradeoffs between FacetMap's graphical approach and the traditional text-oriented approach.

Previous results have shown that users perform better on spatial orientation tasks involving static 2D scenes when working on physically large displays as compared to small ones. This was found to be true even when the displays presented the same images at equivalent visual angles. Further investigation has suggested that large displays may provide a greater sense of presence, which biases users into adopting more efficient strategies to perform tasks. In this work, we extend those findings, demonstrating that users are more effective at performing 3D virtual navigation tasks on large displays. We also show that even though interacting with the environment affects performance, effects induced by interactivity are independent of those induced by physical display size. Together, these findings allow us to derive guidelines for the design and presentation of interactive 3D environments on physically large displays.

Previous research reported interesting gender effects involving specific benefits for females navigating with wider fields of view on large displays. However, it was not clear what was driving the 3D navigation performance gains, and whether or not the effect was more tightly coupled to gender or to spatial abilities. The study we report in this paper replicates and extends previous work, demonstrating that the gender-specific navigation benefits come from the presence of optical flow cues, which are better afforded by wider fields of view on large displays. The study also indicates that the effect may indeed be tied to gender, as opposed to spatial abilities. Together, the findings provide a significant contribution to the HCI community, as we provide strong recommendations for the design and presentation of 3D environments, backed by empirical data. Additionally, these recommendations reliably benefit females, without an accompanying detriment to male navigation performance.

Large wall-sized displays are becoming prevalent. Although researchers have articulated qualitative benefits of group work on large displays, little work has been done to quantify the benefits for individual users. We ran two studies comparing the performance of users working on a large projected wall display to that of users working on a standard desktop monitor. In these studies, we held the visual angle constant by adjusting the viewing distance to each of the displays. Results from the first study indicate that although there was no significant difference in performance on a reading comprehension task, users performed about 26% better on a spatial orientation task done on the large display. Results from the second study suggest that the large display affords a greater sense of presence, allowing users to treat the spatial task as an egocentric rather than an exocentric rotation. We discuss future work to extend our findings and formulate design principles for computer interfaces and physical workspaces.

Published reports suggest that males significantly outperform females in navigating virtual environments. A novel navigation technique reported in CHI 2001, when combined with a large display and wide field of view, appeared to reduce that gender bias. That work has been extended with two navigation studies in order to understand the finding under carefully controlled conditions. The first study replicated the finding that a wide field of view coupled with a large display benefits both male and female users and reduces gender bias. The second study suggested that wide fields of view on a large display were useful to females despite a more densely populated virtual world. Implications for design of virtual worlds and large displays are discussed. Specifically, women take a wider field of view to achieve similar virtual environment navigation performance to men

We present a task-based taxonomy of navigation techniques for 3D virtual environments, used to categorize existing techniques, drive exploration of the design space, and inspire new techniques. We briefly discuss several new techniques, and describe in detail one new techniques, Speed-coupled Flying with Orbiting. This technique couples control of movement speed to camera height and tilt, allowing users to seamlessly transition between local environment-views and global overviews. Users can also orbit specific objects for inspection. Results from two competitive user studies suggest users performed better with Speed-coupled Flying with Orbiting over alternatives, with performance also enhanced by a large display.

In its simplest form, reality is merely information that is presented or acquired. Mixed Reality (MR) is built around the integration of real world physical and computer generated virtual information. We do not use the term augmented reality (AR) because we view the merging of both worlds as a symbiosis, with desirable properties from each accentuated and complementing each other, rather than the enhancement of one with the other. Collaborative MR allows multiple participants to simultaneously share a physical space while being surrounded by a virtual space that is registered with the physical. Because the MR world inherits the properties of real and virtual worlds, it is rich with social context, spatial cues, and tangible objects from the real world as well as flexible digital information from virtual. We believe that Mixed Reality is a medium, largely unexplored, but very well suited for face-to-face collaboration.