Front-end developers author UI components to be broadly reusable by parameterizing visual and behavioral properties. While flexible, this makes instantiation harder, as developers must reason about numerous property values and interactions. In practice, they must explore the component’s large design space and provide realistic and natural values to properties. To address this, we introduce distinguishing variations: variations that are both mimetic and distinct. We frame distinguishing variation generation as design-space sampling, combining symbolic inference to identify visually important properties with an LLM-driven mimetic sampler to produce realistic instantiations from its world knowledge.

Despite being trained on vast amounts of data, most LLMs are unable to reliably generate well-designed UIs. Designer feedback is essential to improving performance on UI generation; however, we find that existing RLHF methods based on ratings or rankings are not well-aligned with with designers' workflows and ignore the rich rationale used to critique and improve UI designs. In this paper, we investigate several approaches for designers to give feedback to UI generation models, using familiar interactions such as commenting, sketching and direct manipulation. We first perform an evaluation with 21 designers where they gave feedback using these interactions, which resulted in 1500 design annotations. We then use this data to finetune a series of LLMs to generate higher quality UIs. Finally, we evaluate these models with human judges, and we find that our designer-aligned approaches outperform models trained with traditional ranking feedback and all tested baselines, including GPT-5.

It is challenging to generate the code for a complete user interface using a Large Language Model (LLM). User interfaces are complex and their implementations often consist of multiple, inter-related files that together specify the contents of each screen, the navigation flows between the screens, and the data model used throughout the application. It is challenging to craft a single prompt for an LLM that contains enough detail to generate a complete user interface, and even then the result is frequently a single large and intricate file that contains all of the generated screens. In this paper, we introduce Athena, a prototype application generation environment that demonstrates how the use of shared intermediate representations, including an app storyboard, data model, and GUI skeletons, can help a developer work with an LLM in an iterative fashion to craft a complete user interface. These intermediate representations also scaffold the LLM’s code generation process, producing organized and structured code in multiple files while limiting errors. We evaluated Athena with a user study with 12 developers. Participants appreciated Athena’s support for prototyping multi-screen iOS apps, acknowledged that the intermediate representations improved their control and understanding of generated code, and discussed the limitations of the system and potential directions for improvement.

Large language model (LLM)-based computer use agents execute user commands by interacting with available UI elements, but little is known about how users want to interact with these agents or what design factors matter for their user experience (UX). We conducted a two-phase study to map the UX design space for computer use agents. In Phase 1, we reviewed existing systems to develop a taxonomy of UX considerations, then refined it through interviews with eight UX and AI practitioners. The resulting taxonomy included categories such as user prompts, explainability, user control, and users’ mental models, with corresponding subcategories and example design features. In Phase 2, we ran a Wizard-of-Oz study with 20 participants, where a researcher acted as a web-based computer use agent and probed user reactions during normal, error-prone and risky execution. We used the findings to validate the taxonomy from Phase 1 and deepen our understand of the design space by identifying the connections between design areas and divergence in user needs and scenarios. Our taxonomy and empirical insights provide a map for developers to consider different aspects of user experience in computer use agent design and to situate their designs within users’ diverse needs and scenarios.

We are again thrilled to be able to present the Computer-Supported Cooperative Work and Social Computing (CSCW) community with an issue of the Proceedings of the ACM on Human-Computer Interaction, containing very interesting and relevant scholarship from its members. This issue includes 42 papers from the May 2025 cycle, selected from a total of 637 submissions and following two rounds of reviews and one revision. 209 submissions from this round will be further revised, reviewed again, and may appear in another issue of the journal later this year. Our external reviewers and track editorial board have together conducted a rigorous review process to select contributions of the highest quality advancing the CSCW field. As Track Chairs, we are grateful for the community’s collective efforts to continue shaping and sharing CSCW’s tradition of high-quality scholarship across the years.

UI prototyping often involves iterating and blending elements from examples such as screenshots and sketches, but current tools offer limited support for incorporating these examples.Inspired by the cognitive process of conceptual blending, we introduce a novel UI workflow that allows developers to rapidly incorporate diverse aspects from design examples into work-in-progress UIs.We prototyped this workflow as Misty.Through a exploratory first-use study with 14 frontend developers, we assessed Misty's effectiveness and gathered feedback on this workflow.Our findings suggest

Building a generalist model for user interface (UI) understanding is challenging due to various foundational issues, such as platform diversity, resolution variation, and data limitation. In this paper, we introduce Ferret-UI 2, a multimodal large language model (MLLM) designed for universal UI understanding across a wide range of platforms, including iPhone, Android, iPad, Webpage, and AppleTV. Building on the foundation of Ferret-UI, Ferret-UI 2 introduces three key innovations: support for multiple platform types, high-resolution perception through adaptive scaling, and advanced task training data generation powered by GPT-4o with set-of-mark visual prompting. These advancements enable Ferret-UI 2 to perform complex, user-centered interactions, making it highly versatile and adaptable for the expanding diversity of platform ecosystems. Extensive empirical experiments on referring, grounding, user-centric advanced tasks (comprising 9 subtasks $\times$ 5 platforms), GUIDE next-action prediction dataset, and GUI-World multi-platform benchmark demonstrate that Ferret-UI 2 significantly outperforms Ferret-UI, and also shows strong cross-platform transfer capabilities.

Publisher Copyright: © 2025 Copyright held by the owner/author(s).

Creating 2D animations is challenging because it requires iterative refinement of movement and transitions across multiple elements within a scene. We explored the potential of LLMs to support animation design by first identifying current challenges in formative interviews with animation creators, and then developing a design probe and LLM-based animation design tool called Keyframer. From user-provided graphics and natural language prompts, Keyframer generates animation code, enables users to preview rendered animations inline, and supports direct edits for iterative design refinement. We utilized this design probe to uncover user prompting styles for describing animation in natural language and observe user strategies for iterating on animations in an exploratory user study with 13 novices and experts in animation design and programming. Through this study, we contribute a categorization of prompting styles users employed for specifying animation goals, along with design insights on supporting iterative refinement of animations through the combination of direct editing and natural language interfaces.

Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs. However, to our knowledge, no one has yet explored the use of LLMs in controlling assistive technologies for the purposes of supporting accessibility testing. In this paper, we explore the requirements of a natural language based accessibility testing workflow, starting with a formative study. From this we build a system that takes a manual accessibility test instruction in natural language (e.g., “Search for a show in VoiceOver”) as input and uses an LLM combined with pixel-based UI Understanding models to execute the test and produce a chaptered, navigable video. In each video, to help QA testers, we apply heuristics to detect and flag accessibility issues (e.g., Text size not increasing with Large Text enabled, VoiceOver navigation loops). We evaluate this system through a 10-participant user study with accessibility QA professionals who indicated that the tool would be very useful in their current work and performed tests similarly to how they would manually test the features. The study also reveals insights for future work on using LLMs for accessibility testing.

Jason Wu, Eldon Schoop, Alan Leung, Titus Barik, Jeffrey Bigham, Jeffrey Nichols. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

User interface (UI) design is a difficult yet important task for ensuring the usability, accessibility, and aesthetic qualities of applications. In our paper, we develop a machine-learned model, UIClip, for assessing the design quality and visual relevance of a UI given its screenshot and natural language description. To train UIClip, we used a combination of automated crawling, synthetic augmentation, and human ratings to construct a large-scale dataset of UIs, collated by description and ranked by design quality. Through training on the dataset, UIClip implicitly learns properties of good and bad designs by i) assigning a numerical score that represents a UI design’s relevance and quality and ii) providing design suggestions. In an evaluation that compared the outputs of UIClip and other baselines to UIs rated by 12 human designers, we found that UIClip achieved the highest agreement with ground-truth rankings. Finally, we present three example applications that demonstrate how UIClip can facilitate downstream applications that rely on instantaneous assessment of UI design quality: i) UI code generation, ii) UI design tips generation, and iii) quality-aware UI example search.

Programmers frequently engage with machine learning tutorials in computational notebooks and have been adopting code generation technologies based on large language models (LLMs). However, they encounter difficulties in understanding and working with code produced by LLMs. To mitigate these challenges, we introduce a novel workflow into computational notebooks that augments LLM-based code generation with an additional ephemeral UI step, offering users UI scaffolds as an intermediate stage between user prompts and code generation. We present this workflow in Biscuit, an extension for JupyterLab that provides users with ephemeral UIs generated by LLMs based on the context of their code and intentions, scaffolding users to understand, guide, and explore with LLMgenerated code. Through a user study where 10 novices used Biscuit for machine learning tutorials, we found that Biscuit offers users representations of code to aid their understanding, reduces the complexity of prompt engineering, and creates a playground for users to explore different variables and iterate on their ideas.

Many apps have basic accessibility issues, like missing labels or low contrast. To supplement manual testing, automated tools can help developers and QA testers find basic accessibility issues, but they can be laborious to use or require writing dedicated tests. To motivate our work, we interviewed eight accessibility QA professionals at a large technology company. From these interviews, we synthesized three design goals for accessibility report generation systems. Motivated by these goals, we developed a system to generate whole app accessibility reports by combining varied data collection methods (e.g., app crawling, manual recording) with an existing accessibility scanner. Many such scanners are based on single-screen scanning, and a key problem in whole app accessibility reporting is to effectively de-duplicate and summarize issues collected across an app. To this end, we developed a screen grouping model with 96.9% accuracy (88.8% F1-score) and UI element matching heuristics with 97% accuracy (98.2% F1-score). We combine these technologies in a system to report and summarize unique issues across an app, and enable a unique pixel-based ignore feature to help engineers and testers better manage reported issues across their app’s lifetime. We conducted a user study where 19 accessibility engineers and testers used multiple tools to create lists of prioritized issues in the context of an accessibility audit. Our system helped them create lists they were more satisfied with while addressing key limitations of current accessibility scanning tools.

Modeling user interfaces (UIs) from visual information allows systems to make inferences about the functionality and semantics needed to support use cases in accessibility, app automation, and testing. Current datasets for training machine learning models are limited in size due to the costly and time-consuming process of manually collecting and annotating UIs. We crawled the web to construct WebUI, a large dataset of 400,000 rendered web pages associated with automatically extracted metadata. We analyze the composition of WebUI and show that while automatically extracted data is noisy, most examples meet basic criteria for visual UI modeling. We applied several strategies for incorporating semantics found in web pages to increase the performance of visual UI understanding models in the mobile domain, where less labeled data is available: (i) element detection, (ii) screen classification and (iii) screen similarity.

Machine learning models have been trained to predict semantic information about user interfaces (UIs) to make apps more accessible, easier to test, and to automate. Currently, most models rely on datasets of static screenshots that are labeled by human annotators, a process that is costly and surprisingly error-prone for certain tasks. For example, workers labeling whether a UI element is “tappable” from a screenshot must guess using visual signifiers, and do not have the benefit of tapping on the UI element in the running app and observing the effects. In this paper, we present the Never-ending UI Learner, an app crawler that automatically installs real apps from a mobile app store and crawls them to infer semantic properties of UIs by interacting with UI elements, discovering new and challenging training examples to learn from, and continually updating machine learning models designed to predict these semantics. The Never-ending UI Learner so far has crawled for more than 5,000 device-hours, performing over half a million actions on 6,000 apps to train three computer vision models for i) tappability prediction, ii) draggability prediction, and iii) screen similarity.

Accurately recognizing icon types in mobile applications is integral to many tasks, including accessibility improvement, UI design search, and conversational agents. Existing research focuses on recognizing the most frequent icon types, but these technologies fail when encountering an unrecognized low-frequency icon. In this paper, we work towards complete coverage of icons in the wild. After annotating a large-scale icon dataset (327,879 icons) from iPhone apps, we found a highly uneven distribution: 98 common icon types covered 92.8% of icons, while 7.2% of icons were covered by more than 331 long-tail icon types. In order to label icons with widely varying occurrences in apps, our system uses an image classification model to recognize common icon types with an average of 3,000 examples each (96.3% accuracy) and applies a few-shot learning model to classify long-tail icon types with an average of 67 examples each (78.6% accuracy). Our system also detects contextual information that helps characterize icon semantics, including nearby text (95.3% accuracy) and modifier symbols added to the icon (87.4% accuracy). In a validation study with workers (n = 23), we verified the usefulness of our generated icon labels. The icon types supported by our work cover 99.5% of collected icons, improving on the previously highest 78% coverage in icon classification work.

All graphical user interfaces are comprised of one or more screens that may be shown to the user depending on their interactions. Identifying different screens of an app and understanding the type of changes that happen on the screens is a challenging task that can be applied in many areas including automatic app crawling, playback of app automation macros and large scale app dataset analysis. For example, an automated app crawler needs to understand if the screen it is currently viewing is the same as any previous screen that it has encountered, so it can focus its efforts on portions of the app that it has not yet explored. Moreover, identifying the type of change on the screen, such as whether any dialogues or keyboards have opened or closed, is useful for an automatic crawler to handle such events while crawling. Understanding screen relationships is a difficult task as instances of the same screen may have visual and structural variation, for example due to different content in a database-backed application, scrolling, dialog boxes opening or closing, or content loading delays. At the same time, instances of different screens from the same app may share some similarities in terms of design, structure, and content. This paper uses a dataset of screenshots from more than 1K iPhone applications to train two ML models that understand similarity in different ways: (1) a screen similarity model that combines a UI object detector with a transformer model architecture to recognize instances of the same screen from a collection of screenshots from a single app, and (2) a screen transition model that uses a siamese network architecture to identify both similarity and three types of events that appear in an interaction trace: the keyboard or a dialog box appearing or disappearing, and scrolling. Our models achieve an F1 score of 0.83 on the screen similarity task, improving on comparable baselines, and an average F1 score of 0.71 across all events in the transition task.

Notifications are an indispensable feature of mobile devices, but their delivery can interrupt and distract users. Prior work has examined interventions, such as deferring notification delivery to opportune moments, but has not systematically studied how users might prefer an intelligent system to manage their notifications. Hence, we directly probed Android smartphone users’ notification preferences via a one-week experience-sampling study ( N = 35). We found that users prefer mitigating undesired interruptions by suppressing alerts over deferring them and referred to notification content factors more frequently than contextual factors for explaining their preferences. Then we demonstrated the challenges and potentials of leveraging user actions to help predict notification preferences. Specifically, we showed that a model personalized using user actions achieved a performance gain of 39% than a generic model. This improvement is similar to the 42% performance gain using labels solicited from the user while using observable user actions causes no extra disruption.

Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring accessibility metadata for mobile apps from their pixels, as the visual interfaces often best reflect an app’s full functionality. We trained a robust, fast, memory-efficient, on-device model to detect UI elements using a dataset of 77,637 screens (from 4,068 iPhone apps) that we collected and annotated. To further improve UI detections and add semantic information, we introduced heuristics (e.g., UI grouping and ordering) and additional models (e.g., recognize UI content, state, interactivity). We built Screen Recognition to generate accessibility metadata to augment iOS VoiceOver. In a study with 9 screen reader users, we validated that our approach improves the accessibility of existing mobile apps, enabling even previously inaccessible apps to be used.

Automated understanding of user interfaces (UIs) from their pixels can improve accessibility, enable task automation, and facilitate interface design without relying on developers to comprehensively provide metadata. A first step is to infer what UI elements exist on a screen, but current approaches are limited in how they infer how those elements are semantically grouped into structured interface definitions. In this paper, we motivate the problem of screen parsing, the task of predicting UI elements and their relationships from a screenshot. We describe our implementation of screen parsing and provide an effective training procedure that optimizes its performance. In an evaluation comparing the accuracy of the generated output, we find that our implementation significantly outperforms current systems (up to 23%). Finally, we show three example applications that are facilitated by screen parsing: (i) UI similarity search, (ii) accessibility enhancement, and (iii) code generation from UI screenshots.

Head-based pointing is an alternative input method for people with motor impairments to access computing devices. This paper proposes a calibration-free head-tracking input mechanism for mobile devices that makes use of the front-facing camera that is standard on most devices. To evaluate our design, we performed two Fitts’ Law studies. First, a comparison study of our method with an existing head-based pointing solution, Eva Facial Mouse, with subjects without motor impairments. Second, we conducted what we believe is the first Fitts’ Law study using a mobile head tracker with subjects with motor impairments. We extend prior studies with a greater range of index of difficulties (IDs) [1.62, 5.20] bits and achieved promising throughput (average 0.61 bps with motor impairments and 0.90 bps without). We found that users’ throughput was 0.95 bps on average in our most difficult task (IDs: 5.20 bits), which involved selecting a target half the size of the Android recommendation for a touch target after moving nearly the full height of the screen. This suggests the system is capable of fine precision tasks. We summarize our observations and the lessons from our user studies into a set of design guidelines for head-based pointing systems.

Sketches and real-world user interface examples are frequently used in multiple stages of the user interface design process. Unfortunately, finding relevant user interface examples, especially in large-scale datasets, is a highly challenging task because user interfaces have aesthetic and functional properties that are only indirectly reflected by their corresponding pixel data and meta-data. This paper introduces Swire, a sketch-based neural-network-driven technique for retrieving user interfaces. We collect the first large-scale user interface sketch dataset from the development of Swire that researchers can use to develop new sketch-based data-driven design interfaces and applications. Swire achieves high performance for querying user interfaces: for a known validation task it retrieves the most relevant example as within the top-10 results for over 60% of queries. With this technique, for the first time designers can accurately retrieve relevant user interface examples with free-form sketches natural to their design workflows. We demonstrate several novel applications driven by Swire that could greatly augment the user interface design process.

Welcome to this issue of the Proceedings of the ACM on Human-Computer Interaction, which will focus on contributions from the research community Engineering Interactive Computing Systems (EICS). This diverse research community explores the methods, processes, techniques and tools that support specifying, designing, developing, deploying and verifying interactive systems. Building interactive systems is a multifaceted and challenging activity, involving a plethora of different actors and roles. This is particularly true in the domain of HCI, where we continuously push the edge of what is possible, where there is a crucial need for adequate processes, tools and methods to build reliable, useful and usable systems that help people cope with the ever-increasing complexity of work and life. The contents of this issue on EICS is the sum of four separate rounds of submissions, evenly spaced from July 2017 through May 2018. In total, the rounds attracted a total of 81 submissions from Asia, Canada, Australia, Europe, Africa, and the United States. Promising submissions in a round that were not accepted were invited to resubmit to a subsequent round, and 6 of the papers appearing in this issue were accepted after at least one round of resubmission. In each round, papers were subject to a rigorous reviewing process where they were reviewed by two EICS senior editors, as well as external reviewers. At the conclusion of each round, a Virtual Committee meeting was held to discuss all of the papers and arrive at final decisions. Ultimately, 14 papers were accepted over all rounds. This issue exists because of the dedicated volunteer effort of 20 senior editors who handled two to four papers each round, and 115 expert reviewers to ensure high quality and insightful reviews for all papers in all rounds. Reviewers and committee members were kept constant as much as possible for papers that were submitted to multiple rounds. Senior members of the editorial group also helped shepherd some papers, reflecting the deep commitment of this research community. We are excited by the detailed and insightful work that resulted in this PACMHCI EICS issue and look forward to equally high quality submissions in subsequent submission cycles over the coming year. For those interested in this area, this group holds their next annual conference June 19-22, 2018 in Paris, France. That conference will provide many opportunities to share ideas with other researchers and practitioners from institutions around the world.

Multi-device product designers need tools to better address ecologically valid constraints in naturalistic settings early in their design process. To address this need, we created a reusable design kit of scenarios, "hint" cards, and a framework that codifies insights from prior work and our own field study. We named the kit the Moving Context Kit, or McKit for short, because it helps designers focus on context shifts that we found to be highly influential in everyday multi-device use. Specifically, we distilled the following findings from our field study in the McKit: (1) devices are typically specialized into one of six roles during parallel use notifier, broadcaster, collector, gamer, remote, and hub, and (2) device roles are influenced by context shifts between private and shared situations. Through a workshop, we validated that the McKit enables designers to engage with complex user needs, situations, and relationships when incorporating novel multi-device techniques into the products they envision.

Data-driven models help mobile app designers understand best practices and trends, and can be used to make predictions about design performance and support the creation of adaptive UIs. This paper presents Rico, the largest repository of mobile app designs to date, created to support five classes of data-driven applications: design search, UI layout generation, UI code generation, user interaction modeling, and user perception prediction. To create Rico, we built a system that combines crowdsourcing and automation to scalably mine design and interaction data from Android apps at runtime. The Rico dataset contains design data from more than 9.7k Android apps spanning 27 categories. It exposes visual, textual, structural, and interactive design properties of more than 72k unique UI screens. To demonstrate the kinds of applications that Rico enables, we present results from training an autoencoder for UI layout similarity, which supports query- by-example search over UIs.

To evaluate the performance of mobile app designs, designers and researchers employ techniques such as A/B, usability, and analytics-driven testing. While these are all useful strategies for evaluating known designs, comparing many divergent solutions to identify the most performant remains a costly and difficult problem. This paper introduces a design performance testing approach that leverages existing app implementations and crowd workers to enable comparative testing at scale. This approach is manifest in ZIPT, a zero-integration performance testing platform that allows designers to collect detailed design and interaction data over any Android app -- including apps they do not own and did not build. Designers can deploy scripted tests via ZIPT to collect aggregate user performance metrics (e.g., completion rate, time on task) and qualitative feedback over third-party apps. Through case studies, we demonstrate that designers can use ZIPT's aggregate data and visualizations to understand the relative performance of interaction patterns found in the wild, and identify usability issues in existing Android apps.

The Proceedings of the ACM (PACM) was initiated by ACM in 2015 as overarching framework for publishing high quality computer science research. The goal for these new journals is to provide an alternate journal publication model for rigorous research papers that have traditionally been presented at major ACM conferences. PACM titles cross multiple intellectual communities, and each separate PACM is designed to represent a broad, but consistent, reach area. This is the first issue of the Proceedings of the ACM on Human Computer Interaction (PACMHCI), which represents the varied topics and communities that compose the broader study of Human Computer Interaction (HCI). The rich heterogeneity of this field can be expressed as deep ethnographies of information use in context, to experiments showing the effectiveness of interface designs, to the production of new technologies that push the limits of how we interact with computers, and much more. The production of PACMHCI will focus on content associated with major research communities that are supported by the ACM Special Interest Group on Human-Computer Interaction (SIGCHI). Individual issues will be largely associated with separate research communities, who may then also select papers from the issue for presentation at their major conferences. These research communities provide the volunteers and editors necessary to provide the rigorous review and editorial process that will define this journal. Those editors will serve on the overall board for ACMHCI, to help bridge our diverse communities. Leveraging our research communities allows us to provide high-quality reviewing while maintaining the quick processing of work that is important in this quickly moving field. This inaugural issue of PACMHCI represents work from the Engineering Interactive Computing Systems (EICS) community. EICS gathers researchers that aim to improve the ways we build interactive systems. Building interactive systems is a multi-faceted and challenging activity, involving a plethora of different actors and roles. This is particularly true in the domain of HCI, where we continuously push the edge of what is possible, where there is a crucial need for adequate processes, tools and methods to build reliable, useful and usable systems that help people cope with the ever-increasing complexity of work and life. The primary goal of the EICS research community is to create novel and high quality contributions in this direction. Although there are only three articles in this first issue, our pipeline for future issues is promising. In the first submission cycle, 41 papers were submitted and, of those, 22 were asked for major revisions. We expect a good number of those 22 papers to be ultimately accepted over the coming months. We are grateful to our newly-formed Editorial Board consisting of more than 70 experts for lending their support and knowledge to the new journal. More information about PACMHCI can be found at http://pacmhci.acm.org/.

research-article Free Access Share on Editorial: End-User Development for the Internet of Things Editors: Panos Markopoulos Eindhoven University of Technology Eindhoven University of TechnologyView Profile , Jeffrey Nichols Google GoogleView Profile , Fabio Paternò CNR-ISTI CNR-ISTIView Profile , Volkmar Pipek University of Siegen University of SiegenView Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 24Issue 2April 2017 Article No.: 9pp 1–3https://doi.org/10.1145/3054765Published:06 April 2017Publication History 19citation867DownloadsMetricsTotal Citations19Total Downloads867Last 12 Months75Last 6 weeks28 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

As the number of computing devices available to users continues to grow, personal computing increasingly involves using multiple devices together. However, support for multi-device interactions has fallen behind users' desire to leverage the diverse capabilities of the devices that surround them. In this paper, we report on an interview study of 29 designers and developers in which we investigate the barriers to creating useful, usable, and delightful multi-device experiences. We uncovered three key challenges: 1) the difficulty in designing the interactions between devices, 2) the complexity of adapting interfaces to different platform UI standards, and 3) the lack of tools and methods for testing multi-device user experiences. We discuss the technological and business factors behind these challenges and potential ways to lower the barriers they impose.

Sporting events evoke strong emotions amongst fans and thus act as natural laboratories to explore emotions and how they unfold in the wild. Computational tools, such as sentiment analysis, provide new ways to examine such dynamic emotional processes. In this article we use sentiment analysis to examine tweets posted during 2014 World Cup. Such analysis gives insight into how people respond to highly emotional events, and how these emotions are shaped by contextual factors, such as prior expectations, and how these emotions change as events unfold over time. Here we report on some preliminary analysis of a World Cup twitter corpus using sentiment analysis techniques. We show these tools can give new insights into existing theories of what makes a sporting match exciting. This analysis seems to suggest that, contrary to assumptions in sports economics, excitement relates to expressions of negative emotion. We also discuss some challenges that such data present for existing sentiment analysis techniques and discuss future analysis.

InkWell is a writer's assistant---a natural language revision program designed to assist creative writers by producing stylistic variations on texts based on craft-based facets of creative writing and by mimicking aspects of specified writers and their personality traits. It is built on top of an optimization process that produces variations on a supplied text, evaluates those variations quantitatively, and selects variations that best satisfy the goals of writing craft and writer mimicry. We describe the design and capabilities of InkWell, and present an early evaluation of its effectiveness and uses with two established literary writers along with an experiment using InkWell to write haiku on its own.

There has been much effort on studying how social media sites, such as Twitter, help propagate information in different situations, including spreading alerts and SOS messages in an emergency. However, existing work has not addressed how to actively identify and engage the right strangers at the right time on social media to help effectively propagate intended information within a desired time frame. To address this problem, we have developed three models: (1) a feature-based model that leverages people's exhibited social behavior, including the content of their tweets and social interactions, to characterize their willingness and readiness to propagate information on Twitter via the act of retweeting; (2) a wait-time model based on a user's previous retweeting wait times to predict his or her next retweeting time when asked; and (3) a subset selection model that automatically selects a subset of people from a set of available people using probabilities predicted by the feature-based model and maximizes retweeting rate. Based on these three models, we build a recommender system that predicts the likelihood of a stranger to retweet information when asked, within a specific time window, and recommends the top-N qualified strangers to engage with. Our experiments, including live studies in the real world, demonstrate the effectiveness of our work.

This paper presents an experiment on the relationship between personal values and reading interests of online articles. Results suggest that individuals' values can predict their topical interests. For example, holding stronger universalism values predict interests towards environmental articles, whereas holding stronger achievement values predict interest towards work-related articles. Findings demonstrate the possibility of targeting based on individuals' personal values, but also highlight certain challenges and limitations when applying this approach for online content.

The use of social media at work is gaining traction, and there is evidence to suggest that various benefits accrue from its use. Yet the relationship between using social media at work and employee performance is not clear. Through a study of 75,747 employees of a large global company over the course of 3 years, we find that some social media usage (number of forum posts, forum post length, and status update length) was positively associated with performance ratings. This study is one of the first to show the relationship among different forms of social media use and employee performance ratings.

Massive amounts of data are being generated on social media sites, such as Twitter and Facebook. These data can be used to better understand people (e.g., personality traits, perceptions, and preferences) and predict their behavior. As a result, a deeper understanding of users and their behavior can benefit a wide range of intelligent applications, such as advertising, social recommender systems, and personalized knowledge management. These applications will also benefit individual users themselves and optimize their experience across a wide variety of domains, such as retail, healthcare, and education. Since mining and understanding user behavior from social media often requires interdisciplinary effort, including machine learning, text mining, human-computer interaction, and social science, our workshop aims to bring together researchers and practitioners from multiple fields to discuss the creation of deeper models of individual users by mining the content that they publish and the social networking behavior that they exhibit.

The theory of values posits that each person has a set of values, or desirable and trans-situational goals, that motivate their actions. The Basic Human Values, a motivational construct that captures people's values, have been shown to influence a wide range of human behaviors. In this work, we analyze people's values and their word use on Reddit, an online social news sharing community. Through conducting surveys and analyzing text contributions of 799 Reddit users, we identify and interpret categories of words that are indicative of user's value orientations. Using the same data, we further report a preliminary exploration on word-based prediction of Basic Human Values.

There has been much effort on studying how social media sites, such as Twitter, help propagate information in different situations, including spreading alerts and SOS messages in an emergency. However, existing work has not addressed how to actively identify and engage the right strangers at the right time on social media to help effectively propagate intended information within a desired time frame. To ad-dress this problem, we have developed two models: (i) a feature-based model that leverages peoplesfi exhibited social behavior, including the content of their tweets and social interactions, to characterize their willingness and readiness to propagate information on Twitter via the act of retweeting; and (ii) a wait-time model based on a user's previous retweeting wait times to predict her next retweeting time when asked. Based on these two models, we build a recommender system that predicts the likelihood of a stranger to retweet information when asked, within a specific time window, and recommends the top-N qualified strangers to engage with. Our experiments, including live studies in the real world, demonstrate the effectiveness of our work.

This paper presents a system, System U, which automatically derives people's personality traits from social media and recommends people for different tasks. The system leverages linguistic signals appearing in a person's social media activities to compute the personality portraits including Big Five personality, fundamental needs and basic human values. This system and technology can be used in a wide variety of personalized applications, such as recommending people to answer questions.

We present a new algorithm for inferring the home location of Twitter users at different granularities, including city, state, time zone, or geographic region, using the content of users’ tweets and their tweeting behavior. Unlike existing approaches, our algorithm uses an ensemble of statistical and heuristic classifiers to predict locations and makes use of a geographic gazetteer dictionary to identify place-name entities. We find that a hierarchical classification approach, where time zone, state, or geographic region is predicted first and city is predicted next, can improve prediction accuracy. We have also analyzed movement variations of Twitter users, built a classifier to predict whether a user was travelling in a certain period of time, and use that to further improve the location detection accuracy. Experimental evidence suggests that our algorithm works well in practice and outperforms the best existing algorithms for predicting the home location of Twitter users.

Massive amounts of data are being generated on social media sites, such as Twitter and Facebook. These data can be used to better understand people (e.g., personality traits, perceptions, and preferences) and predict their behavior. As a result, a deeper understanding of users and their behavior can benefit a wide range of intelligent applications, such as advertising, social recommender systems, and personalized knowledge management. These applications will also benefit individual users themselves and optimize their experience across a wide variety of domains, such as retail, healthcare, and education. Since mining and understanding user behavior from social media often requires interdisciplinary effort, including machine learning, text mining, human-computer interaction, and social science, our workshop aims to bring together researchers and practitioners from multiple fields to discuss the creation of deeper models of individual users by mining the content that they publish and the social networking behavior that they exhibit.

An online community consists of a group of users who share a common interest, background, or experience and their collective goal is to contribute towards the welfare of the community members. Question answering is an important feature that enables community members to exchange knowledge within the community boundary. The overwhelming number of communities necessitates the need for a good question routing strategy so that new questions gets routed to the appropriately focused community and thus get resolved. In this paper, we consider the novel problem of routing questions to the right community and propose a framework to select the right set of communities for a question. We begin by using several prior proposed features for users and add some additional features, namely language attributes and inclination to respond, for community modeling. Then we introduce two k nearest neighbor based aggregation algorithms for computing community scores. We show how these scores can be combined to recommend communities and test the effectiveness of the recommendations over a large real world dataset.

The emergence of social media creates a unique opportunity for developing a new class of crowd-powered information collection systems. Such systems actively identify potential users based on their public social media posts and solicit them directly for information. While studies have shown that users will respond to solicitations in a few domains, there is little analysis of the quality of information received. Here we explore the quality of information solicited from Twitter users in the domain of product reviews, specifically reviews for a popular tablet computer and L.A.-based food trucks. Our results show that the majority of responses to our questions (>70%) contained relevant information and often provided additional details (>37%) beyond the topic of the question. We compare the solicited Twitter reviews to other user-generated reviews from Amazon and Yelp, and found that the Twitter answers provided similar information when controlling for the questions asked. Our results also reveal limitations of this new information collection method, including its suitability in certain domains and potential technical barriers to its implementation. Our work provides strong evidence for the potential of this new class of information collection systems and design implications for their future use.

We present an intelligent, crowd-powered information collection system that automatically identifies and asks targeted strangers on Twitter for desired information (e.g., current wait time at a nightclub). Our work includes three parts. First, we identify a set of features that characterize one's willingness and readiness to respond based on their exhibited social behavior, including the content of their tweets and social interaction patterns. Second, we use the identified features to build a statistical model that predicts one's likelihood to respond to information solicitations. Third, we develop a recommendation algorithm that selects a set of targeted strangers using the probabilities computed by our statistical model with the goal to maximize the over-all response rate. Our experiments, including several in the real world, demonstrate the effectiveness of our work.

Despite decades of research attempting to establish conversational interaction between humans and computers, the capabilities of automated conversational systems are still limited. In this paper, we introduce Chorus, a crowd-powered conversational assistant. When using Chorus, end users converse continuously with what appears to be a single conversational partner. Behind the scenes, Chorus leverages multiple crowd workers to propose and vote on responses. A shared memory space helps the dynamic crowd workforce maintain consistency, and a game-theoretic incentive mechanism helps to balance their efforts between proposing and voting. Studies with 12 end users and 100 crowd workers demonstrate that Chorus can provide accurate, topical responses, answering nearly 93% of user queries appropriately, and staying on-topic in over 95% of responses. We also observed that Chorus has advantages over pairing an end user with a single crowd worker and end users completing their own tasks in terms of speed, quality, and breadth of assistance. Chorus demonstrates a new future in which conversational assistants are made usable in the real world by combining human and machine intelligence, and may enable a useful new way of interacting with the crowds powering other systems.

Task automation systems promise to increase human productivity by assisting us with our mundane and difficult tasks. These systems often rely on people to (1) identify the tasks they want automated and (2) specify the procedural steps necessary to accomplish those tasks (i.e., to create task models). However, our interviews with users of a Web task automation system reveal that people find it difficult to identify tasks to automate and most do not even believe they perform repetitive tasks worthy of automation. Furthermore, even when automatable tasks are identified, the well-recognized difficulties of specifying task steps often prevent people from taking advantage of these automation systems. In this research, we analyze real Web usage data and find that people do in fact repeat behaviors on the Web and that automating these behaviors, regardless of their complexity, would reduce the overall number of actions people need to perform when completing their tasks, potentially saving time. Motivated by these findings, we developed LiveAction, a fully-automated approach to generating task models from Web usage data. LiveAction models can be used to populate the task model repositories required by many automation systems, helping us take advantage of automation in our everyday lives.

Massive amounts of data are being generated on social media sites, such as Twitter and Facebook. This data can be used to better understand people, such as their personality traits, perceptions, and preferences, and predict their behavior. This deeper understanding of users and their behaviors can benefit a wide range of intelligent applications, such as advertising, social recommender systems, and personalized knowledge management. These applications will also benefit individual users themselves by optimizing their experiences across a wide variety of domains, such as retail, healthcare, and education. Since mining and understanding user behavior from social media often requires interdisciplinary effort, including machine learning, text mining, human-computer interaction, and social science, our workshop aims to bring together researchers and practitioners from multiple fields to discuss the creation of deeper models of individual users by mining the content that they publish and the social networking behavior that they exhibit.

In spite of decades of research on virtual worlds, our understanding of one popular form of virtual world behavior - raiding - remains limited. Raiding is important because it entails intense, high-risk, and complex collaborative behaviors in computer-mediated environments. This paper contributes to CSCW literature by offering a longitudinal analysis of raiding behavior using system data manually collected from the game world itself, comparing two raiding teams as they worked through the same content. Supplemented with interviews and chat transcripts, this research sheds light on what actually happens during raids across four different temporal scales: seconds, hours, days, and months. It also distinguishes between behaviors that are imposed by the system design and those chosen by players. Finally, it derives two viable raiding styles from the data.

When people have questions, they often turn to their social network for answers. If the answer is obscure or time sensitive however, no members of their social networks may know the answer. For example, it may be difficult to find a friend who has experience with a particular feature or model of digital camera or who knows the current wait time for security at the local airport. In this paper, we explore the feasibility of answering questions by asking strangers. In this approach, strangers with potentially useful information are identified by mining the public status updates posted on Twitter, questions are sent to these strangers, and responses are collected. We explore feasibility in two ways: will users respond to questions sent by strangers and, if they do respond, how long must we wait for a response? Our results from asking 1159 questions across two domains suggest that 42% of users will respond to questions from strangers. 44% of these responses arrived within 30 minutes.

Massive amounts of data are being generated on social media sites, such as Twitter and Facebook. People from all walks of life share data about social events, express opinions, discuss their interests, publicize businesses, recommend products, and, explicitly or implicitly, reveal personal information. This workshop will focus on the use of social media data for creating models of individual users from the content that they publish. Deeper understanding of user behavior and associated attributes can benefit a wide range of intelligent applications, such as social recommender systems and expert finders, as well as provide the foundation in support of novel user interfaces (e.g., actively engaging the crowd in mixed-initiative question-answering systems). These applications and interfaces may offer significant benefits to users across a wide variety of domains, such as retail, government, healthcare and education. User modeling from public social media data may also reveal information that users would prefer to keep private. Such concerns are particularly important because individuals do not have complete control over the information they share about themselves. For example, friends of a user may inadvertently divulge private information about that user in their own posts. In this workshop we will also discuss possible mechanisms that users might employ to monitor what information has been revealed about themselves on social media and obfuscate any sensitive information that has been accidentally revealed.

The status updates posted to social networks, such as Twitter and Facebook, contain a myriad of information about what people are doing and watching. During events, such as sports games, many updates are sent describing and expressing opinions about the event. In this paper, we describe an algorithm that generates a journalistic summary of an event using only status updates from Twitter as a source. Temporal cues, such as spikes in the volume of status updates, are used to identify the important moments within an event, and a sentence ranking method is used to extract relevant sentences from the corpus of status updates describing each important moment within an event. We evaluate our algorithm compared to human-generated summaries and the previous best summarization algorithm, and find that the results of our method are superior to the previous algorithm and approach the readability and grammaticality of the human-generated summaries.

ActionShot is an integrated web browser tool that creates a fine-grained history of users' browsing activities by continually recording their browsing actions at the level of interactions, such as button clicks and entries into form fields. ActionShot provides interfaces to facilitate browsing and searching through this history, sharing portions of the history through established social networking tools such as Facebook, and creating scripts that can be used to repeat previous interactions at a later time. ActionShot can also create short textual summaries for sequences of interactions. In this paper, we describe the ActionShot and our initial explorations of the tool through field deployments within our organization and a lab study. Overall, we found that ActionShot's history features provide value beyond typical browser history interfaces.

This paper presents CoCo, a system that automates web tasks on a user's behalf through an interactive conversational interface. Given a short command such as "get road conditions for highway 88," CoCo synthesizes a plan to accomplish the task, executes it on the web, extracts an informative response, and returns the result to the user as a snippet of text. A novel aspect of our approach is that we leverage a repository of previously recorded web scripts and the user's personal web browsing history to determine how to complete each requested task. This paper describes the design and implementation of our system, along with the results of a brief user study that evaluates how likely users are to understand what CoCo does for them.

Collaborative web browsing tasks occur frequently, such as one user showing another how to use a web site, several users working together on a search task, or even one user sending an interesting link to another user. Unfortunately, tools for browsing the web are commonly designed for a single user. PlayByPlay is a general purpose web collaboration tool that uses the communication model of instant messaging to support a variety of collaborative browsing tasks. PlayByPlay also supports collaborative browsing between mobile and desktop users, which we believe is useful for on-the-go scenarios. We conducted user studies of the desktop and mobile versions of PlayByPlay and found the system to be usable and effective.

For blind web users, completing tasks on the web can be frustrating. Each step can require a time-consuming linear search of the current web page to find the needed interactive element or piece of information. Existing interactive help systems and the playback components of some programming-by-demonstration tools identify the needed elements of a page as they guide the user through predefined tasks, obviating the need for a linear search on each step. We introduce TrailBlazer, a system that provides an accessible, non-visual interface to guide blind users through existing how-to knowledge. A formative study indicated that participants saw the value of TrailBlazer but wanted to use it for tasks and web sites for which no existing script was available. To address this, TrailBlazer offers suggestion-based help created on-the-fly from a short, user-provided task description and an existing repository of how-to knowledge. In an evaluation on 15 tasks, the correct prediction was contained within the top 5 suggestions 75.9% of the time.

Mashups are an increasingly popular way to integrate data from multiple web sites to fit a particular need, but it often requires substantial technical expertise to create them. To lower the barrier for creating mashups, we have extended the CoScripter web automation tool with a spreadsheet-like environment called Vegemite. Our system uses direct-manipulation and programming-by-demonstration tech-niques to automatically populate tables with information collected from various web sites. A particular strength of our approach is its ability to augment a data set with new values computed by a web site, such as determining the driving distance from a particular location to each of the addresses in a data set. An informal user study suggests that Vegemite may enable a wider class of users to address their information needs.

The deep web contains an order of magnitude more information than the surface web, but that information is hidden behind the web forms of a large number of web sites. Metasearch engines can help users explore this information by aggregating results from multiple resources, but previously these could only be created and maintained by programmers. In this paper, we explore the automatic creation of metasearch mash-ups by mining the web interactions of multiple web users to find relations between query forms on different web sites. We also present an implemented system called TX2 that uses those connections to search multiple deep web resources simultaneously and integrate the results in context in a single results page. TX2 illustrates the promise of constructing mash-ups automatically and the potential of mining web interactions to explore deep web resources.

Over six years, we iterated on the design of a language for describing the functionality of appliances, such as televisions, telephones, VCRs, and copiers. This language has been used to describe more than thirty diverse appliances, and these descriptions have been used to automatically generate both graphical and speech user interfaces on handheld computers, mobile phones, and desktop computers. In this article, we describe the final design of our language and analyze the key design choices that led to this design. Through this analysis, we hope to provide a useful guide for the designers of future user interface description languages.

Today's web pages provide many useful features, but unfortunately nearly all are designed first and foremost for the desktop form factor. At the same time, the number of mobile devices with different form factors and unique input and output facilities is growing substantially. The Highlight environment addresses these problems by allowing users to start with existing sites they already use and create mobile versions that are customized to their tasks and mobile devices. This re-authoring is performed through a combination of demonstrating desired interactions with an existing web site and directly specifying content to be included on mobile pages. The system has been tested successfully with a variety of existing sites. A study showed that novice users were able to use the system to create useful mobile applications for sites of their own choosing.

We present a new server-side architecture that enables rapid prototyping and deployment of mobile web applications created from existing web sites. Key to this architecture is a remote control metaphor in which the mobile device controls a fully functional browser that is embedded within a proxy server. Content is clipped from the proxy browser, transformed if necessary, and then sent to the mobile device as a typical web page. Users' interactions with that content on the mobile device control the next steps of the proxy browser. We have found this approach to work well for creating mobile sites from a variety of existing sites, including those that use dynamic HTML and AJAX technologies. We have conducted a small user study to evaluate our model and API with experienced web programmers.

Users increasingly interact with a heterogeneous collection of computing devices. The applications that users employ on those devices, however, still largely provide user experiences that assume the use of a single computer. This failure is due in part to the difficulty of creating user experiences that span multiple devices, particularly the need to manage identifying, connecting to, and communicating with other devices. In this paper we present an infrastructure based on instant messaging that simplifies adding that additional functionality to applications. Our infrastructure elevates device ownership to a first class property, allowing developers to provide functionality that spans personal devices without writing code to manage users' devices or establish connections among them. It also provides simple mechanisms for applications to send information, events, or commands between a user's devices. We demonstrate the effectiveness of our infrastructure by presenting a set of sample applications built with it and a user study demonstrating that developers new to the infrastructure can implement all of the cross-device functionality for three applications in, on average, less than two and a half hours.

We conducted a user study that demonstrates that automatically generated interfaces can support better usability through increased flexibility in two dimensions. First, we show that automatic generation can improve usability by moving interfaces that are constrained by cost and poor interaction primitives to another device with better interactive capabilities: subjects were twice as fast and four times as successful at completing tasks with automatically generated interfaces on a PocketPC device as with the actual appliance interfaces. Second, we show that an automatic generator can improve usability by automatically ensuring that new interfaces are generated to be consistent with users' previous experience: subjects were also twice as fast using interfaces consistent with their experiences as compared to normally generated interfaces. These two results demonstrate that automatic interface generation is now viable and especially desirable where users will benefit from individualized interfaces or where human designers are constrained by cost and other factors.

A problem with many of today's appliance interfaces is that they are inconsistent. For example, the procedure for setting the time on alarm clocks and VCRs differs, even among different models made by the same manufacturer. Finding particular functions can also be a challenge, because appliances often organize their features differently. This paper presents a system, called Uniform, which approaches this problem by automatically generating remote control interfaces that take into account previous interfaces that the user has seen during the generation process. Uniform is able to automatically identify similarities between different devices and users may specify additional similarities. The similarity information allows the interface generator to use the same type of controls for similar functions, place similar functions so that they can be found with the same navigation steps, and create interfaces that have a similar visual appearance.

Systems of connected appliances, such as home theaters and presentation rooms, are becoming commonplace in our homes and workplaces. These systems are often difficult to use, in part because users must determine how to split the tasks they wish to perform into sub-tasks for each appliance and then find the particular functions of each appliance to complete their sub-tasks. This paper describes Huddle, a new system that automatically generates task-based interfaces for a system of multiple appliances based on models of the content flow within the multi-appliance system.

One of the challenges of using mobile devices for ubiquitous re-mote control is the creation of the user interface. If automatically generated designs are used, then they must be close in quality to hand-designed interfaces. Automatically generated interfaces can be dramatically improved if they use standard conventions to which users are accustomed, such as the arrangement of buttons on a telephone dial-pad or the conventional play, stop, and pause icons on a media player. Unfortunately, it can be difficult for a system to determine where to apply design conventions because each appliance may represent its functionality differently. Smart Templates is a technique that uses parameterized templates in the appliance model to specify when such conventions might be automatically applied in the user interface. Our templates easily adapt to existing appliance models and interface generators on different platforms can apply appropriate design conventions using templates.

People with Muscular Dystrophy (MD) and certain other muscular and nervous system disorders lose their gross motor control while retaining fine motor control. The result is that they lose the ability to move their wrists and arms, and therefore their ability to operate a mouse and keyboard. However, they can often still use their fingers to control a pencil or stylus, and thus can use a handheld computer such as a Palm. We have developed software that allows the handheld to substitute for the mouse and keyboard of a PC, and tested it with four people (ages 10, 12, 27 and 53) with MD. The 12-year old had lost the ability to use a mouse and keyboard, but with our software, he was able to use the Palm to access email, the web and computer games. The 27-year-old reported that he found the Palm so much better that he was using it full-time instead of a keyboard and mouse. The other two subjects said that our software was much less tiring than using the conventional input devices, and enabled them to use computers for longer periods. We report the results of these case studies, and the adaptations made to our software for people with disabilities.

It is difficult to interact with computer displays that are across the room. A popular approach is to use laser pointers tracked by a camera, but interaction techniques using laser pointers tend to be imprecise, error-prone, and slow. Although many previous papers discuss laser pointer interaction techniques, none seem to have performed user studies to help inform the design. This paper reports on two studies of laser pointer interactions that answer some of the questions related to interacting with objects using a laser pointer. The first experiment evaluates various parameters of laser pointers. For example, the time to acquire a target is about 1 second, and the jitter due to hand unsteadiness is about ±8 pixels, which can be reduced to about ±2 to ±4 pixels by filtering. We compared 7 different ways to hold various kinds of laser pointers, and found that a laser pointer built into a PalmOS device was the most stable. The second experiment compared 4 different ways to select objects on a large projected display. We found that tapping directly on a wall-size SmartBoard was the fastest and most accurate method, followed by a new interaction technique that copies the area of interest from the big screen to a handheld. Third in speed was the conventional mouse, and the laser pointer came in last, with a time almost twice as long as tapping on the SmartBoard

We describe our system which facilitates collaboration using multiple modalities, including speech, handwriting, gestures, gaze tracking, direct manipulation, large projected touch-sensitive displays, laser pointer tracking, regular monitors with a mouse and keyboard, and wireless networked handhelds. Our system allows multiple, geographically dispersed participants to simultaneously and flexibly mix different modalities using the right interface at the right time on one or more machines. We discuss each of the modalities provided, how they were integrated in the system architecture, and how the user interface enabled one or more people to flexibly use one or more devices.

Several industrial and academic research groups are working to simplify the control of appliances and services by creating a truly universal remote control. Unlike the preprogrammed remote controls available today, these new controllers download a specification from the appliance or service and use it to automatically generate a remote control interface. This promises to be a useful approach because the specification can be made detailed enough to generate both speech and graphical interfaces. Unfortunately, generating good user interfaces can be difficult. Based on user studies and prototype implementations, this paper presents a set of requirements that we have found are needed for automatic interface generation systems to create high-quality user interfaces.

The personal universal controller (PUC) is an approach for improving the interfaces to complex appliances by introducing an intermediary graphical or speech interface. A PUC engages in two-way communication with everyday appliances, first downloading a specification of the appliance's functions, and then automatically creating an interface for controlling that appliance. The specification of each appliance includes a high-level description of every function, a hierarchical grouping of those functions, and dependency information, which relates the availability of each function to the appliance's state. Dependency information makes it easier for designers to create specifications and helps the automatic interface generators produce a higher quality result. We describe the architecture that supports the PUC, and the interface generators that use our specification language to build high-quality graphical and speech interfaces.