Users can adapt contemporary natural language interfaces (NLIs) by teaching the NLIs how to handle new natural language (NL) inputs. One promising approach is interactive task learning (ITL), which enables users to teach new NL inputs for multi-modal systems. While recent advances enable users to teach the syntactic and semantic level of the NL inputs through ITL, NLIs are still not able to learn how to consider the context, such as the current state of the graphical user interface (GUI). To address this challenge, we designed MALACHITE through three formative studies. MALACHITE enables users to successfully teach NL inputs on a semantic and syntactic level leveraging the GUI screen of a data visualization tool. With two evaluative studies, we provide evidence that with MALACHITE ’s suggestions, users significantly improve their accuracy by a factor of 2.3 in teaching GUI-dependent NL inputs in contrast to those without MALACHITE ’s suggestions.

In the early 2000s, software development was shifting from offline to online, and from command line to IDE. We discuss our 2004 paper examining the impact of this shift on developers’ program comprehension behaviors, our motivation for the work, and it's impact on the last twenty years of empirical studies and developer tool innovations. We end with a discussion of the possible unintended impacts LLMs have on program comprehension in the coming decades.

Modern software engineering is in a state of flux. With more development utilizing AI code generation tools and the continued reliance on online programming resources, understanding code and the original intent behind it is becoming more important than it ever has been. To this end, we have developed the “Meta-Manager”, a Visual Studio Code extension, with a supplementary browser extension, that automatically collects and organizes changes made to code while keeping track of the provenance of each part of the code, including code that has been AI-generated or copy-pasted from popular programming resources online. These sources and subsequent changes are represented in the editor and may be explored using searching and filtering mechanisms to help developers answer historically hard-to-answer questions about code, its provenance, and its design rationale. In our evaluation of Meta-Manager, we found developers were successfully able to use it to answer otherwise unanswerable questions about an unfamiliar code base.

Sensemaking in unfamiliar domains can be challenging, demanding considerable user effort to compare different options with respect to various criteria. Prior research and our formative study found that people would benefit from reading an overview of an information space upfront, including the criteria others previously found useful. However, existing sensemaking tools struggle with the “cold-start” problem — it not only requires significant input from previous users to generate and share these overviews, but such overviews may also turn out to be biased and incomplete. In this work, we introduce a novel system, Selenite, which leverages Large Language Models (LLMs) as reasoning machines and knowledge retrievers to automatically produce a comprehensive overview of options and criteria to jumpstart users’ sensemaking processes. Subsequently, Selenite also adapts as people use it, helping users find, read, and navigate unfamiliar information in a systematic yet personalized manner. Through three studies, we found that Selenite produced accurate and high-quality overviews reliably, significantly accelerated users’ information processing, and effectively improved their overall comprehension and sensemaking experience.

Almost no modern software system is written from scratch, and developers are required to effectively learn to use third-party libraries and software services. Thus, many practitioners and researchers have looked for ways to create effective documentation that supports developers’ learning. However, few efforts have focused on how people actually use the documentation. In this paper, we report on an exploratory, multi-phase, mixed methods empirical study of documentation page-view logs from four cloud-based industrial services. By analyzing page-view logs for over 100,000 users, we find diverse patterns of documentation page visits. Moreover, we show statistically that which documentation pages people visit often correlates with user characteristics such as past experience with the specific product, on the one hand, and with future adoption of the API on the other hand. We discuss the implications of these results on documentation design and propose documentation page-view log analysis as a feasible technique for design audits of documentation, from ones written for software developers to ones designed to support end users (e.g., Adobe Photoshop).

Mixed reality (MR) provides users with novel affordances that allow them to overlay and experience reality in various visual manifestations. However, existing works mainly focused on using MR to augment a user’s present, which does not exhaust the full potential of contextual MR. In this paper, we empirically explore MR-Driven Near-Future Realities, a future multimodal MR experience that can overlay semantically-related augmentations within a user’s everyday life to allow them to preview, manipulate, and reflect on a near-future reality. We investigated this concept during a VR-based empirical study to understand users’ perceptions and opinions about MR-Driven Near-Future Realities. The results showed that users were positive about the concept of MR-Driven Near-Future Realities and were capable of eliciting near-future realities in various simulated real-world manifestations, but more demanding scenarios negatively affected their experience and task completion performance. Our goal is to spark meaningful and critical discussions about the use of MR-Driven Near-Future Realities to preview possible near-future realities in everyday life.

The software engineering community recently has witnessed widespread deployment of AI programming assistants, such as GitHub Copilot. However, in practice, developers do not accept AI programming assistants' initial suggestions at a high frequency. This leaves a number of open questions related to the usability of these tools. To understand developers' practices while using these tools and the important usability challenges they face, we administered a survey to a large population of developers and received responses from a diverse set of 410 developers. Through a mix of qualitative and quantitative analyses, we found that developers are most motivated to use AI programming assistants because they help developers reduce key-strokes, finish programming tasks quickly, and recall syntax, but resonate less with using them to help brainstorm potential solutions. We also found the most important reasons why developers do not use these tools are because these tools do not output code that addresses certain functional or non-functional requirements and because developers have trouble controlling the tool to generate the desired output. Our findings have implications for both creators and users of AI programming assistants, such as designing minimal cognitive effort interactions with these tools to reduce distractions for users while they are programming.

Understanding code is challenging, especially when working in new and complex development environments. Code comments and documentation can help, but are typically scarce or hard to navigate. Large language models (LLMs) are revolutionizing the process of writing code. Can they do the same for helping understand it? In this study, we provide a first investigation of an LLM-based conversational UI built directly in the IDE that is geared towards code understanding. Our IDE plugin queries OpenAI's GPT-3.5-turbo model with four high-level requests without the user having to write explicit prompts: to explain a highlighted section of code, provide details of API calls used in the code, explain key domain-specific terms, and provide usage examples for an API. The plugin also allows for open-ended prompts, which are automatically contextualized to the LLM with the program being edited. We evaluate this system in a user study with 32 participants, which confirms that using our plugin can aid task completion more than web search. We additionally provide a thorough analysis of the ways developers use, and perceive the usefulness of, our system, among others finding that the usage and benefits differ between students and professionals. We conclude that in-IDE prompt-less interaction with LLMs is a promising future direction for tool builders.

Adaptive user interfaces (AUIs) can improve user experience by automatically adapting how information and functionality are presented in a user interface. However, the dynamic nature and potentially numerous variations of AUIs make them challenging to author. In this paper, we present a generalized framework for defining adaptation as interpolations between UIs and introduce a computational approach for intelligently generating new variations of a UI from a small set of designs. Based on this approach, we develop FrameKit, an authoring tool with a programming-by-example interface that retains flexibility and control afforded by manual authoring while reducing effort through automatic generation. We demonstrate that FrameKit can support adaptations that typically require domain-specific toolkits, such as those found in context-aware applications, responsive UIs, and ability-based adaptation. We evaluated FrameKit with ten front-end developers, who successfully authored AUIs after a short tutorial session and suggested that FrameKit provides an effective mental model for AUI authoring.

Users are increasingly empowered to personalize natural language interfaces (NLIs) by teaching how to handle new natural language (NL) inputs. However, our formative study found that when teaching new NL inputs, users require assistance in clarifying ambiguities that arise and want insight into which parts of the input the NLI understands. In this paper we introduce ONYX, an intelligent agent that interactively learns new NL inputs by combining NL programming and programming-by-demonstration, also known as multi-modal interactive task learning. To address the aforementioned challenges, ONYX provides suggestions on how ONYX could handle new NL inputs based on previously learned concepts or user-defined procedures, and poses follow-up questions to clarify ambiguities in user demonstrations, using visual and textual aids to clarify the connections. Our evaluation shows that users provided with ONYX’s new features achieved significantly higher accuracy in teaching new NL inputs (median: 93.3%) in contrast to those without (median: 73.3%).

Developers constantly learn new APIs, but often lack necessary information from documentation, resorting instead to popular question-and-answer platforms such as Stack Overflow. In this paper, we investigate how to use recent machine-Iearning-based knowledge extraction techniques to automatically identify pairs of comparable API methods and the sentences describing the comparison from Stack Overflow answers. We first built a prototype that can be stocked with a dataset of comparable API methods and provides tool-tips to users in search results and in API documentation. We conducted a user study with this tool based on a dataset of TensorFlow comparable API methods spanning 198 hand-annotated facts from Stack Overflow posts. This study confirmed that providing comparable API methods can be useful for helping developers understand the design space of APIs: developers using our tool were significantly more aware of the comparable API methods and better understood the differences between them. We then created SOREL, an comparable API methods knowledge extraction tool trained on our hand-annotated corpus, which achieves a 71% precision and 55% recall at discovering our manually extracted facts and discovers 433 pairs of comparable API methods from thousands of unseen Stack Overflow posts. This work highlights the merit of jointly studying programming assistance tools and constructing machine learning techniques to power them.

In the United States alone, software testing labor is estimated to cost $48 billion USD per year. Despite widespread test execution automation and automation in other areas of software engineering, test suites continue to be created manually by software engineers. We have built a test generation tool, called NaNofuzz, that helps users find bugs in their code by suggesting tests where the output is likely indicative of a bug, e.g., that return NaN (not-a-number) values. NaNofuzz is an interactive tool embedded in a development environment to fit into the programmer's workflow. NaNofuzz tests a function with as little as one button press, analyses the program to determine inputs it should evaluate, executes the program on those inputs, and categorizes outputs to prioritize likely bugs. We conducted a randomized controlled trial with 28 professional software engineers using NaNofuzz as the intervention treatment and the popular manual testing tool, Jest, as the control treatment. Participants using NaNofuzz on average identified bugs more accurately (p < .05, by 30%), were more confident in their tests (p < .03, by 20%), and finished their tasks more quickly (p < .007, by 30%).

When creating a software project of any significant size or complexity, developers must write about the code in some form. While this is a common practice, tooling support for creating, and perhaps more significantly, maintaining these documents remains limited, despite these documents benefiting later users of the corresponding code. In order to combat some of these known challenges, we developed Sodalite – a context-aware, Visual Studio Code extension for writing and maintaining code-related documents. Sodalite represents a holistic approach to the documentation authoring, maintaining, and using life cycle by suggesting relevant code to attach to the document using common code-writing templates, evaluating how up-to-date the document is using the code in the editor, automatically attempting to identify out-of-date information, and, based on how successful the system is, notifying the user of how trustworthy the document seems to be given their version of the related code. In our preliminary evaluation of Sodalite, we found that users of the system were able to successfully author documents about their code and the system's out-of-date link checking was accurate 86.5% of the time.

A key goal of software engineering research is to improve the environments, tools, languages, and techniques programmers use to efficiently create quality software. Successfully designing these tools and demonstrating their effectiveness involves engaging with tool users—software engineers. Researchers often want to conduct user studies of software engineers to collect direct evidence. However, running user studies can be difficult, and researchers may lack solution strategies to overcome the barriers, so they may avoid user studies. To understand the challenges researchers face when conducting programmer user studies, we interviewed 26 researchers. Based on the analysis of interview data, we contribute (i) a taxonomy of 18 barriers researchers encounter; (ii) 23 solution strategies some researchers use to address 8 of the 18 barriers in their own studies; and (iii) 4 design ideas, which we adapted from the behavioral science community, that may lower 8 additional barriers. To validate the design ideas, we held an in-person all-day focus group with 16 researchers.

Modern software development requires developers to find and effectively utilize new APIs and their documentation, but documentation has many well-known issues. Despite this, developers eventually overcome these issues but have no way of sharing what they learned. We investigate sharing this documentation-specific information through annotations, which have advantages over developer forums as the information is contextualized, not disruptive, and is short, thus easy to author. Developers can also author annotations to support their own comprehension. In order to support the documentation usage behaviors we found, we built the Adamite annotation tool, which provides features such as multiple anchors, annotation types, and pinning. In our user study, we found that developers are able to create annotations that are useful to themselves and are able to utilize annotations created by other developers when learning a new API, with readers of the annotations completing 67% more of the task, on average, than the baseline.

Developers perform online sensemaking on a daily basis, such as researching and choosing libraries and APIs. Prior research has introduced tools that help developers capture information from various sources and organize it into structures useful for subsequent decision-making. However, it remains a laborious process for developers to manually identify and clip content, maintaining its provenance and synthesizing it with other content. In this work, we introduce a new system called Crystalline that automatically collects and organizes information into tabular structures as the user searches and browses the web. It leverages natural language processing to automatically group similar criteria together to reduce clutter, and uses passive behavioral signals such as mouse movement and dwell time to infer what information to collect and how to visualize and prioritize it. Our user study suggests that developers are able to create comparison tables about 20% faster with a 60% reduction in operational cost without sacrificing the quality of the tables.

Developers spend significant amounts of time finding, relating, navigating, and, more broadly, making sense of code. While sensemaking, developers must keep track of many pieces of information including the objectives of their task, the code locations of interest, their questions and hypotheses about the behavior of the code, and more. Despite this process being such an integral aspect of software development, there is little tooling support for externalizing and keeping track of developers’ information, which led us to develop Catseye – an annotation tool for lightweight notetaking about code. Catseye has advantages over traditional methods of externalizing code-related information, such as commenting, in that the annotations retain the original context of the code while not actually modifying the underlying source code, they can support richer interactions such as lightweight versioning, and they can be used as navigational aids. In our investigation of developers’ notetaking processes using Catseye, we found developers were able to successfully use annotations to support their code sensemaking when completing a debugging task.

Consumers conducting comparison shopping, researchers making sense of competitive space, and developers looking for code snippets online all face the challenge of capturing the information they find for later use without interrupting their current flow. In addition, during many learning and exploration tasks, people need to externalize their mental context, such as estimating how urgent a topic is to follow up on, or rating a piece of evidence as a “pro” or “con,” which helps scaffold subsequent deeper exploration. However, current approaches incur a high cost, often requiring users to select, copy, context switch, paste, and annotate information in a separate document without offering specific affordances that capture their mental context. In this work, we explore a new interaction technique called “wiggling,” which can be used to fluidly collect, organize, and rate information during early sensemaking stages with a single gesture. Wiggling involves rapid back-and-forth movements of a pointer or up-and-down scrolling on a smartphone, which can indicate the information to be collected and its valence, using a single, light-weight gesture that does not interfere with other interactions that are already available. Through implementation and user evaluation, we found that wiggling helped participants accurately collect information and encode their mental context with a 58% reduction in operational cost while being 24% faster compared to a common baseline.

Representing the semantics of GUI screens and components is crucial to data-driven computational methods for modeling user-GUI interactions and mining GUI designs. Existing GUI semantic representations are limited to encoding either the textual content, the visual design and layout patterns, or the app contexts. Many representation techniques also require significant manual data annotation efforts. This paper presents Screen2Vec, a new self-supervised technique for generating representations in embedding vectors of GUI screens and components that encode all of the above GUI features without requiring manual annotation using the context of user interaction traces. Screen2Vec is inspired by the word embedding method Word2Vec, but uses a new two-layer pipeline informed by the structure of GUIs and interaction traces and incorporates screen- and app-specific metadata. Through several sample downstream tasks, we demonstrate Screen2Vec’s key useful properties: representing between-screen similarity through nearest neighbors, composability, and capability to represent user tasks.

Despite the increasing complexity and scale of people’s online activities, browser interfaces have stayed largely the same since tabs were introduced in major browsers nearly 20 years ago. The gap between simple tab-based browser interfaces and the complexity of users’ tasks can lead to serious adverse effects – commonly referred to as “tab overload.” This paper introduces a Chrome extension called Tabs.do, which explores bringing a task-centric approach to the browser, helping users to group their tabs into tasks and then organize, prioritize, and switch between those tasks fluidly. To lower the cost of importing, Tabs.do uses machine learning to make intelligent suggestions for grouping users’ open tabs into task bundles by exploiting behavioral and semantic features. We conducted a field deployment study where participants used Tabs.do with their real-life tasks in the wild, and showed that Tabs.do can decrease tab clutter, enabled users to create rich task structures with lightweight interactions, and allowed participants to context-switch among tasks more efficiently.

As the amount of information online continues to grow, a correspondingly important opportunity is for individuals to reuse knowledge which has been summarized by others rather than starting from scratch. However, appropriate reuse requires judging the relevance, trustworthiness, and thoroughness of others' knowledge in relation to an individual's goals and context. In this work, we explore augmenting judgements of the appropriateness of reusing knowledge in the domain of programming, specifically of reusing artifacts that result from other developers' searching and decision making. Through an analysis of prior research on sensemaking and trust, along with new interviews with developers, we synthesized a framework for reuse judgements. The interviews also validated that developers express a desire for help with judging whether to reuse an existing decision. From this framework, we developed a set of techniques for capturing the initial decision maker's behavior and visualizing signals calculated based on the behavior, to facilitate subsequent consumers' reuse decisions, instantiated in a prototype system called Strata. Results of a user study suggest that the system significantly improves the accuracy, depth, and speed of reusing decisions. These results have implications for systems involving user-generated content in which other users need to evaluate the relevance and trustworthiness of that content.

Programming language design requires making many usability-related design decisions. However, existing HCI methods can be impractical to apply to programming languages: languages have high iteration costs, programmers require significant learning time, and user performance has high variance. To address these problems, we adapted both formative and summative HCI methods to make them more suitable for programming language design. We integrated these methods into a new process, PLIERS, for designing programming languages in a user-centered way. We assessed PLIERS by using it to design two new programming languages. Glacier extends Java to enable programmers to express immutability properties effectively and easily. Obsidian is a language for blockchains that includes verification of critical safety properties. Empirical studies showed that the PLIERS process resulted in languages that could be used effectively by many programmers and revealed additional opportunities for language improvement.

A major problem in task-oriented conversational agents is the lack of support for the repair of conversational breakdowns. Prior studies have shown that current repair strategies for these kinds of errors are often ineffective due to: (1) the lack of transparency about the state of the system's understanding of the user's utterance; and (2) the system's limited capabilities to understand the user's verbal attempts to repair natural language understanding errors. This paper introduces SOVITE, a new multi-modal speech plus direct manipulation interface that helps users discover, identify the causes of, and recover from conversational breakdowns using the resources of existing mobile app GUIs for grounding. SOVITE displays the system's understanding of user intents using GUI screenshots, allows users to refer to third-party apps and their GUI screens in conversations as inputs for intent disambiguation, and enables users to repair breakdowns using direct manipulation on these screenshots. The results from a remote user study with 10 users using SOVITE in 7 scenarios suggested that SOVITE's approach is usable and effective.

An important concern in end user development (EUD) is accidentally embedding personal information in program artifacts when sharing them. This issue is particularly important in GUI-based programming-by-demonstration (PBD) systems due to the lack of direct developer control of script contents. Prior studies reported that these privacy concerns were the main barrier to script sharing in EUD. We present a new approach that can identify and obfuscate the potential personal information in GUI-based PBD scripts based on the uniqueness of information entries with respect to the corresponding app GUI context. Compared with the prior approaches, ours supports broader types of personal information beyond explicitly pre-specified ones, requires minimal user effort, addresses the threat of re-identification attacks, and can work with third-party apps from any task domain. Our approach also recovers obfuscated fields locally on the script consumer's side to preserve the shared scripts' transparency, readability, robustness, and generalizability. Our evaluation shows that our approach (1) accurately identifies the potential personal information in scripts across different apps in diverse task domains; (2) allows end-user developers to feel comfortable sharing their own scripts; and (3) enables script consumers to understand the operation of shared scripts despite the obfuscated fields.

Some blockchain programs (smart contracts) have included serious security vulnerabilities. Obsidian is a new typestate-oriented programming language that uses a strong type system to rule out some of these vulnerabilities. Although Obsidian was designed to promote usability to make it as easy as possible to write programs, strong type systems can cause a language to be difficult to use. In particular, ownership, typestate, and assets, which Obsidian uses to provide safety guarantees, have not seen broad adoption together in popular languages and result in significant usability challenges. We performed an empirical study with 20 participants comparing Obsidian to Solidity, which is the language most commonly used for writing smart contracts today. We observed that Obsidian participants were able to successfully complete more of the programming tasks than the Solidity participants. We also found that the Solidity participants commonly inserted asset-related bugs, which Obsidian detects at compile time.

Blockchain platforms are coming into use for processing critical transactions among participants who have not established mutual trust. Many blockchains are programmable, supporting smart contracts , which maintain persistent state and support transactions that transform the state. Unfortunately, bugs in many smart contracts have been exploited by hackers. Obsidian is a novel programming language with a type system that enables static detection of bugs that are common in smart contracts today. Obsidian is based on a core calculus, Silica, for which we proved type soundness. Obsidian uses typestate to detect improper state manipulation and uses linear types to detect abuse of assets. We integrated a permissions system that encodes a notion of ownership to allow for safe, flexible aliasing. We describe two case studies that evaluate Obsidian’s applicability to the domains of parametric insurance and supply chain management, finding that Obsidian’s type system facilitates reasoning about high-level states and ownership of resources. We compared our Obsidian implementation to a Solidity implementation, observing that the Solidity implementation requires much boilerplate checking and tracking of state, whereas Obsidian does this work statically.

Data scientists are responsible for the analysis decisions they make, but it is hard for them to track the process by which they achieved a result. Even when data scientists keep logs, it is onerous to make sense of the resulting large number of history records full of overlapping variants of code, output, plots, etc. We developed algorithmic and visualization techniques for notebook code environments to help data scientists forage for information in their history. To test these interventions, we conducted a think-aloud evaluation with 15 data scientists, where participants were asked to find specific information from the history of another person's data science project. The participants succeed on a median of 80% of the tasks they performed. The quantitative results suggest promising aspects of our design, while qualitative results motivated a number of design improvements. The resulting system, called Verdant, is released as an open-source extension for JupyterLab.

Multi-touch gestures can be very difficult to program correctly because they require that developers build high-level abstractions from low-level touch events. In this paper, we introduce programming primitives that enable programmers to implement multi-touch gestures in a more understandable way by helping them build these abstractions. Our design of these primitives was guided by a formative study, in which we observed developers' natural implementations of custom gestures. Touch groups provide summaries of multiple fingers rather than requiring that programmers track them manually. Cross events allow programmers to summarize the movement of one or a group of fingers. We implemented these two primitives in two environments: a declarative programming system and in a standard imperative programming language. We found that these primitives are capable of defining nuanced multi-touch gestures, which we illustrate through a series of examples. Further, in two user evaluations of these programming primitives, we found that multi-touch behaviors implemented in these programming primitives are more understandable than those implemented with standard touch events.

Designing usable APIs is critical to developers' productivity and software quality, but is quite difficult. One of the challenges is that anticipating API usability barriers and real-world usage is difficult, due to a lack of automated approaches to mine usability data at scale. In this paper, we focus on one particular grievance that developers repeatedly express in online discussions about APIs: "boilerplate code." We investigate what properties make code count as boilerplate, the reasons for boilerplate, and how programmers can reduce the need for it. We then present MARBLE, a novel approach to automatically mine boilerplate code candidates from API client code repositories. MARBLE adapts existing techniques, including an API usage mining algorithm, an AST comparison algorithm, and a graph partitioning algorithm. We evaluate MARBLE with 13 Java APIs, and show that our approach successfully identifies both already-known and new API-related boilerplate code instances.

Natural language programming is a promising approach to enable end users to instruct new tasks for intelligent agents. However, our formative study found that end users would often use unclear, ambiguous or vague concepts when naturally instructing tasks in natural language, especially when specifying conditionals. Existing systems have limited support for letting the user teach agents new concepts or explaining unclear concepts. In this paper, we describe a new multi-modal domain-independent approach that combines natural language programming and programming-by-demonstration to allow users to first naturally describe tasks and associated conditions at a high level, and then collaborate with the agent to recursively resolve any ambiguities or vagueness through conversations and demonstrations. Users can also define new procedures and concepts by demonstrating and referring to contents within GUIs of existing mobile apps. We demonstrate this approach in PUMICE, an end-user programmable agent that implements this approach. A lab study with 10 users showed its usability.

Developers spend a significant portion of their time searching for solutions and methods online. While numerous tools have been developed to support this exploratory process, in many cases the answers to developers' questions involve trade-offs among multiple valid options and not just a single solution. Through interviews, we discovered that developers express a desire for help with decision-making and understanding trade-offs. Through an analysis of Stack Overflow posts, we observed that many answers describe such trade-offs. These findings suggest that tools designed to help a developer capture information and make decisions about trade-offs can provide crucial benefits for both the developers and others who want to understand their design rationale. In this work, we probe this hypothesis with a prototype system named Unakite that collects, organizes, and keeps track of information about trade-offs and builds a comparison table, which can be saved as a design rationale for later use. Our evaluation results show that Unakite reduces the cost of capturing tradeoff-related information by 45%, and that the resulting comparison table speeds up a subsequent developer's ability to understand the trade-offs by about a factor of three.

Almost all software development revolves around the discovery and use of application programming interfaces (APIs). Once a suitable API is selected, programmers must begin the process of determining what functionality in the API is relevant to a programmer's task and how to use it. Our work aims to understand how API functionality is discovered by programmers and where tooling may be appropriate. We employed a mixed-methods approach to investigate Apache Beam, a distributed data processing API, by mining Beam client code and running a lab study to see how people discover Beam's available functionality. We found that programmers' prior experience with similar APIs significantly impacted their ability to find relevant features in an API and attempting to form a top-down mental model of an API resulted in less discovery of features.

Literate programming tools are used by millions of programmers today, and are intended to facilitate presenting data analyses in the form of a narrative. We interviewed 21 data scientists to study coding behaviors in a literate programming environment and how data scientists kept track of variants they explored. For participants who tried to keep a detailed history of their experimentation, both informal and formal versioning attempts led to problems, such as reduced notebook readability. During iteration, participants actively curated their notebooks into narratives, although primarily through cell structure rather than markdown explanations. Next, we surveyed 45 data scientists and asked them to envision how they might use their past history in an future version control system. Based on these results, we give design guidance for future literate programming tools, such as providing history search based on how programmers recall their explorations, through contextual details including images and parameters.

Programmers consistently engage in cognitively demanding tasks such as sense making and decision-making. During the information-foraging process, programmers are growing more reliant on resources available online since they contain masses of crowdsourced information and are easier to navigate. Content available in questions and answers on Stack Overflow presents a unique platform for studying the types of problems encountered in programming and possible solutions. In addition to classifying these questions, we introduce possible visual representations for organizing the gathered information and propose that such models may help reduce the cost of navigating, understanding and choosing solution alternatives.

Experimentation through code is central to data scientists' work. Prior work has identified the need for interaction techniques for quickly exploring multiple versions of the code and the associated outputs. Yet previous approaches that provide history information have been challenging to scale: real use produces a high number of versions of different code and non-code artifacts with dependency relationships and a convoluted mix of different analysis intents. Prior work has found that navigating these records to pick out the relevant information for a given task is difficult and time consuming. We introduce Verdant, a new system with a novel versioning model to support fast retrieval and sensemaking of messy version data. Verdant provides light-weight interactions for comparing, replaying, and tracing relationships among many versions of different code and non-code artifacts in the editor. We implemented Verdant into Jupyter Notebooks, and validated the usability of Verdant's interactions through a usability study.

A key challenge for generalizing programming-by-demonstration (PBD) scripts is the data description problem - when a user demonstrates performing an action, the system needs to determine features for describing this action and the target object in a way that can reflect the user's intention for the action. However, prior approaches for creating data descriptions in PBD systems have problems with usability, applicability, feasibility, transparency and/or user control. Our APPINITE system introduces a multimodal interface with which users can specify data descriptions verbally using natural language instructions. APPINITE guides users to describe their intentions for the demonstrated actions through mixed-initiative conversations. APPINITE constructs data descriptions for these actions from the natural language instructions. Our evaluation showed that APPINITE is easy-to-use and effective in creating scripts for tasks that would otherwise be difficult to create with prior PBD systems, due to ambiguous data descriptions in demonstrations on GUIs.

Application Programming Interfaces (APIs) are a rapidly growing industry and the usability of the APIs is crucial to programmer productivity. Although prior research has shown that APIs commonly suffer from significant usability problems, little attention has been given to studying how APIs are designed and created in the first place. We interviewed 24 professionals involved with API design from 7 major companies to identify their training and design processes. Interviewees had insights into many different aspects of designing for API usability and areas of significant struggle. For example, they learned to do API design on the job, and had little training for it in school. During the design phase they found it challenging to discern which potential use cases of the API users will value most. After an API is released, designers lack tools to gather aggregate feedback from this data even as developers openly discuss the API online.

Though conditionals are an integral component of programming, providing an easy means of creating conditionals remains a challenge for programming-by-demonstration (PBD) systems for task automation. We hypothesize that a promising method for implementing conditionals in such systems is to incorporate the use of verbal instructions. Verbal instructions supplied concurrently with demonstrations have been shown to improve the generalizability of PBD. However, the challenge of supporting conditional creation using this multi-modal approach has not been addressed. In this extended abstract, we present our study on understanding how end users describe conditionals in natural language for mobile app tasks. We conducted a formative study of 56 participants asking them to verbally describe conditionals in different settings for 9 sample tasks and to invent conditional tasks. Participant responses were analyzed using open coding and revealed that, in the context of mobile apps, end users often omit desired else statements when explaining conditionals, sometimes use ambiguous concepts in expressing conditionals, and often desire to implement complex conditionals. Based on these findings, we discuss the implications for designing a multimodal PBD interface to support the creation of conditionals.

How do people ideate through code? Using semi-structured interviews and a survey, we studied data scientists who program, often with small scripts, to experiment with data. These studies show that data scientists frequently code new analysis ideas by building off of their code from a previous idea. They often rely on informal versioning interactions like copying code, keeping unused code, and commenting out code to repurpose older analysis code while attempting to keep those older analyses intact. Unlike conventional version control, these informal practices allow for fast versioning of any size code snippet, and quick comparisons by interchanging which versions are run. However, data scientists must maintain a strong mental map of their code in order to distinguish versions, leading to errors and confusion. We explore the needs for improving version control tools for exploratory tasks, and demonstrate a tool for lightweight local versioning, called Variolite, which programmers found usable and desirable in a preliminary usability study.

SUGILITE is a new programming-by-demonstration (PBD) system that enables users to create automation on smartphones. SUGILITE uses Android's accessibility API to support automating arbitrary tasks in any Android app (or even across multiple apps). When the user gives verbal commands that SUGILITE does not know how to execute, the user can demonstrate by directly manipulating the regular apps' user interface. By leveraging the verbal instructions, the demonstrated procedures, and the apps? UI hierarchy structures, SUGILITE can automatically generalize the script from the recorded actions, so SUGILITE learns how to perform tasks with different variations and parameters from a single demonstration. Extensive error handling and context checking support forking the script when new situations are encountered, and provide robustness if the apps change their user interface. Our lab study suggests that users with little or no programming knowledge can successfully automate smartphone tasks using SUGILITE.

Though immutability has been long-proposed as a way to prevent bugs in software, little is known about how to make immutability support in programming languages effective for software engineers. We designed a new formalism that extends Java to support transitive class immutability, the form of immutability for which there is the strongest empirical support, and implemented that formalism in a tool called Glacier. We applied Glacier successfully to two real-world systems. We also compared Glacier to Java's final in a user study of twenty participants. We found that even after being given instructions on how to express immutability with final, participants who used final were unable to express immutability correctly, whereas almost all participants who used Glacier succeeded. We also asked participants to make specific changes to immutable classes and found that participants who used final all incorrectly mutated immutable state, whereas almost all of the participants who used Glacier succeeded. Glacier represents a promising approach to enforcing immutability in Java and provides a model for enforcement in other languages.

In open-ended tasks where a program's behavior cannot be specified in advance, exploratory programming is a key practice in which programmers actively experiment with different possibilities using code. Exploratory programming is highly relevant today to a variety of professional and end-user programmer domains, including prototyping, learning through play, digital art, and data science. However, prior research has largely lacked clarity on what exploratory programming is, and what behaviors are characteristic of this practice. Drawing on this data and prior literature, we provide an organized description of what exploratory programming has meant historically and a framework of four dimensions for studying exploratory programming tasks: (1) applications, (2) required code quality, (3) ease or difficulty of exploration, and (4) the exploratory process. This provides a basis for better analyzing tool support for exploratory programming.

Moonstone is a new plugin for Eclipse that supports developers in understanding exception flow and in writing exception handlers in Java. Understanding exception control flow is paramount for writing robust exception handlers, a task many developers struggle with. To help with this understanding, we present two new kinds of information: ghost comments, which are transient overlays that reveal potential sources of exceptions directly in code, and annotated highlights of skipped code and associated handlers. To help developers write better handlers, Moonstone additionally provides project-specific recommendations, detects common bad practices, such as empty or inadequate handlers, and provides automatic resolutions, introducing programmers to advanced Java exception handling features, such as try-with-resources. We present findings from two formative studies that informed the design of Moonstone. We then show with a user study that Moonstone improves users' understanding in certain areas and enables developers to amend exception handling code more quickly and correctly.

We designed, implemented, and evaluated a small physical keyboard, composed of four to five keys with a variety of mappings for the letters, along with one or two navigational function keys operated by the thumb. The small size allows it to be used for smartwatch text entry, and can be mounted on various body parts as a wearable keyboard. It is primarily used as an ambiguous keyboard, akin to T9, although a multi-tap mode is also present. Our keyboard layouts include Alphabetic, Collapsed QWERTY, Mnemonic (where letters are grouped based on their shapes), and Optimized (which minimizes the number of conflicts). In a between-users study, participants achieved an average of 15.4 words per minute (WPM) across all layouts, with one user with the Collapsed QWERTY layout reaching a top speed of 30.6 WPM after 4 hours of practice. User feedback was generally favorable.

Our paper on “Mica” from the 2006 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCCO6 [1] presented a prototype tool that helped programmers use a description of desired functionality to find specific relevant API classes and methods. Mica worked by using the Google Web APIs to find pages relating to a given query, and then using a list of known API classes and methods to recognize and aggregate the API-relevant words contained in the result pages. Mica used weighted word frequencies to find the API-relevant words correlated with the search and displayed the most relevant words alongside regular web search results. We had observed that when programmers create new software or add functionality to existing software, a common and often difficult task is that of using libraries, toolkits and SDKs to achieve the desired functionality. Internet search engines, in particular Google, have emerged as new and effective tools for this task, providing quick access to a large collection of tutorials and example code. However, our observations of how programmers used these resources revealed problems and inefficiencies in their use as programming tools. Mica was the first of long series of tools that we built as part of the “Natural Programming Project” [2] to help programmers more easily use APIs. We speculate that the impact of the Mica paper is because: ·The problem of helping users find and use the right API from a growing number of available options has become increasingly relevant. ·The data that comes from other developers' experience, found and ranked through web search, continues to be an invaluable starting point for finding and figuring out how to use APIs. · The strategy of extracting and displaying relevant information alongside Google search results has proven effective. Since Mica came out Google and Bing have started to support this pattern for movies, people, events, and many other pieces of information, although not yet APIs. ·That sweet logo!

More and more data nowadays exist in hierarchical formats such as JSON due to the increasing popularity of web applications and web services. While many end-user systems support getting hierarchical data from databases without programming, they provide very little support for using hierarchical data beyond turning the data into a flat string or table. In this paper, we present a spreadsheet tool for using and exploring hierarchical datasets. We introduce novel interaction techniques and algorithms to manipulate and visualize hierarchical data in a spreadsheet using the data's relative hierarchical relationships with the data in its adjacent columns. Our tool leverages the data's structural information to support selecting, grouping, joining, sorting and filtering hierarchical data in spreadsheets. Our lab study showed that our tool helped spreadsheet users complete data exploration tasks nearly two times faster than using Excel and even outperform programmers in most tasks.

Context: Critical software systems developed for the government continue to be of lower quality than expected, despite extensive literature describing best practices in software engineering. Goal: We wanted to better understand the extent of certain issues in the field and the relationship to software quality. Method: We surveyed fifty software development professionals and asked about practices and barriers in the field and the resulting software quality. Results: There is evidence of certain problematic issues for developers and specific quality characteristics that seem to be affected. Conclusions: This motivates future work to address the most problematic barriers and issues impacting software quality.

Programming languages can restrict state change by preventing it entirely (immutability) or by restricting which clients may modify state (read-only restrictions). The benefits of immutability and read-only restrictions in software structures have been long-argued by practicing software engineers, researchers, and programming language designers. However, there are many proposals for language mechanisms for restricting state change, with a remarkable diversity of techniques and goals, and there is little empirical data regarding what practicing software engineers want in their tools and what would benefit them. We systematized the large collection of techniques used by programming languages to help programmers prevent undesired changes in state. We interviewed expert software engineers to discover their expectations and requirements, and found that important requirements, such as expressing immutability constraints, were not reflected in features available in the languages participants used. The interview results informed our design of a new language extension for specifying immutability in Java. Through an iterative, participatory design process, we created a tool that reflects requirements from both our interviews and the research literature.

Many have argued that the current try/catch mechanism for handling exceptions in Java is flawed. A major complaint is that programmers often write minimal and low quality handlers. We used the Boa tool to examine a large number of Java projects on GitHub to provide empirical evidence about how programmers currently deal with exceptions. We found that programmers handle exceptions locally in catch blocks much of the time, rather than propagating by throwing an Exception. Programmers make heavy use of actions like Log, Print, Return, or Throw in catch blocks, and also frequently copy code between handlers. We found bad practices like empty catch blocks or catching Exception are indeed widespread. We discuss evidence that programmers may misjudge risk when catching Exception, and face a tension between handlers that directly address local program statement failure and handlers that consider the program-wide implications of an exception. Some of these issues might be addressed by future tools which autocomplete more complete handlers.

We present a spreadsheet model for working with streaming data. Our prototype tool presents techniques to let the user stream data from web services and web input elements to a spreadsheet without preprogramming those sources into the tool. Spreadsheet cells record metadata about where and when the data came from, allowing the user to view and manipulate streaming data using temporal information. Starting and pausing a data stream in the spreadsheet can be controlled programmatically using values computed by spreadsheet cells, making the spreadsheet program highly dynamic and interactive. We demonstrate the range of our design with a series of examples highlighting its ability to create different kinds of applications that process real-time data from the web using simple spreadsheet formulas.

Today's widely deployed painting applications use a linear undo model that allows users to backtrack previous operations in reverse chronological order. This undo model is not useful if the user has performed desired operations after undesired ones. Selective undo, in contrast, allows users to select specific operations in the past and only undo those, while keeping the remaining operations intact. Although selective undo has been widely explored in the context of text editing and object-oriented drawing, we explore selective undo for painting (bitmap) editing, which has received less attention and introduces many interesting user interface design challenges. Our system, called Aquamarine, explores the script model for selective undo, where selectively undone operations are skipped in the history, rather than the more explored inverse model, which puts an inverse of the selected operations at the end of the history. We discuss the design implications and show through two informal user studies that selective undo is usable and desirable

Programmers often need to revert some code to an earlier state, or restore a block of code that was deleted a while ago. However, support for this backtracking in modern programming environments is limited. Many of the backtracking tasks can be accomplished by having a selective undo feature in code editors, but this has major challenges: there can be conflicts among edit operations, and it is difficult to provide usable interfaces for selective undo. In this paper, we present AZURITE, an Eclipse plug-in that allows programmers to selectively undo fine-grained code changes made in the code editor. With AZURITE, programmers can easily perform backtracking tasks, even when the desired code is not in the undo stack or a version control system. AZURITE also provides novel user interfaces specifically designed for selective undo, which were iteratively improved through user feedback gathered from actual users in a preliminary field trial. A formal lab study showed that programmers can successfully use AZURITE, and were twice as fast as when limited to conventional features.

Cilk Plus and OpenMP are parallel language extensions for the C and C++ programming languages. The CPLEX Study Group of the ISO/IEC C Standards Committee is developing a proposal for a parallel programming extension to C that combines ideas from Cilk Plus and OpenMP. We conducted a preliminary comparison of Cilk Plus and OpenMP in a master's level course on security to evaluate the design tradeoffs in the usability and security of these two approaches. The eventual goal is to inform decision-making within the committee. We found several usability problems worthy of further investigation based on student performance, including declaring and using reductions, multi-line compiler directives, and the understanda-bility of task assignment to threads.

Previously, we presented our technique for visualizing fine-grained code changes in a timeline view, designed to facilitate reviewing and interacting with the code change history. During user evaluations, it became evident that users often wanted to see the code changes at a higher level of abstraction. Therefore, we developed a novel approach to automatically summarize fine-grained code changes into more conceptual, higher-level changes in real time. Our system provides four collapse levels, which are integrated with the timeline via semantic zooming: raw level (no collapsing), statement level, method level, and type level. Compared to the raw level, the number of code changes shown in the timeline at each level is reduced by 55%, 77%, and 83%, respectively. This implies that the semantic zooming would help users better understand and interact with the history by minimizing the potential information overload.

As educational games have become a larger field of study, there has been a growing need for analytic methods that can be used to assess game design and inform iteration. While much previous work has focused on the measurement of student engagement or learning at a gross level, we argue that new methods are necessary for measuring the alignment of a game to its target learning goals at an appropriate level of detail to inform design decisions. We present a novel technique that we have employed to examine alignment in an open-ended educational game. The approach is based on examining how the game reacts to representative student solutions that do and do not obey target principles. We demonstrate this method using real student data and discuss how redesign might be informed by these techniques.

While more and more data are available through web services, it remains difficult for end-users to create web applications that make use of these data without having to write complex code. We present Gneiss, a live programming environment that extends the spreadsheet metaphor to support creating interactive web applications that dynamically use local or web data from multiple sources. Gneiss closely integrates a spreadsheet editor with a web interface builder to let users demonstrate bindings between properties of web GUI elements and cells in the spreadsheet while working with real web service data. The spreadsheet editor provides two-way connections to web services, to both visualize and retrieve different data based on the user input in the web interface. Gneiss achieves rich interactivity without the need for event-based programming by extending the 'pull model' of formulas that is familiar to the spreadsheet users. We use a series of examples to demonstrate Gneiss's ability to create a variety of interactive web data applications.

InterState is a new programming language and environment that addresses the challenges of writing and reusing user interface code. InterState represents interactive behaviors clearly and concisely using a combination of novel forms of state machines and constraints. It also introduces new language features that allow programmers to easily modularize and reuse behaviors. InterState uses a new visual notation that allows programmers to better understand and navigate their code. InterState also includes a live editor that immediately updates the running application in response to changes in the editor and vice versa to help programmers understand the state of their program. Finally, InterState can interface with code and widgets written in other languages, for example to create a user interface in InterState that communicates with a database. We evaluated the understandability of InterState's programming primitives in a comparative laboratory study. We found that participants were twice as fast at understanding and modifying GUI components when they were implemented with InterState than when they were implemented in a conventional textual event-callback style. We evaluated InterState's scalability with a series of benchmarks and example applications and found that it can scale to implement complex behaviors involving thousands of objects and constraints.

Web services offer a more reliable and efficient way to access online data than scraping web pages. However, web service data are often in complex hierarchical structures that make it difficult for people to extract the desired parts or to perform any further data manipulation without writing a significant amount of surprisingly intricate code. In this paper, we present Gneiss, a tool that extends the familiar spreadsheet metaphor to support working with data returned from web services. Gneiss allows users to extract the desired fields in web service data using drag-and-drop, and refine the results through spreadsheet formulas, along with sorting and filtering the data. Hierarchical data are stored as nested tables in the spreadsheet and can be flattened for future operations. Data flow is two-way between the spreadsheet and the web services, enabling people to easily make a new request by modifying spreadsheet cells. In addition, using the dependency between spreadsheet cells, our tool is able to create parallel-running data extractions based on the user's sequential demonstration. We use a set of examples to demonstrate our tool's ability to create fast and reusable data extraction and manipulation programs that work with complex web service data.

Programming often involves reverting source code to an earlier state, which we call backtracking. We performed a longitudinal study of programmers' backtracking, analyzing 1,460 hours of fine-grained code editing logs collected from 21 people. Our analysis method keeps track of the change history of each abstract syntax tree node and looks for backtracking instances within each node. Using this method, we detected a total of 15,095 backtracking instances, which gives an average backtracking rate of 10.3/hour. The size of backtracking varied con-siderably, ranging from a single character to thousands of char-acters. 34% of the backtracking was performed by manually deleting or typing the desired code, and 9.5% of all backtracking was selective, meaning that it could not have been performed using the conventional undo command present in the IDE. The study results show that programmers need better backtracking tools, and also provide design implications for such tools.

Programmers often need to backtrack, but backtracking support in modern programming environments is limited. Previously, we have conducted a series of studies, which discovered that backtracking in programming is in fact prevalent and programmers need better backtracking tools. In this demonstration, we will present our backtracking tool called AZURITE, which provides selective undo and history search and visualization features in the Eclipse code editor. The demonstration will include the user interface presented in our previous work, as well as the new features added in response to user feedback.

The field of Educational Games has seen many calls for added rigor. One avenue for improving the rigor of the field is developing more generalizable methods for measuring student learning within games. Throughout the process of development, what is relevant to measure and assess may change as a game evolves into a finished product. The field needs an approach for game developers and researchers to be able to prototype and experiment with different measures that can stand up to rigorous scrutiny, as well as provide insight into possible new directions for development. We demonstrate a toolkit and analysis tools that capture and analyze students' performance within open educational games. The system records relevant events during play, which can be used for analysis of player learning by designers. The tools support replaying student sessions within the original game's environment, which allows researchers and developers to explore possible explanations for student behavior. Using this system, we were able to facilitate a number of analyses of student learning in an open educational game developed by a team of our collaborators as well as gain greater insight into student learning with the game and where to focus as we iterate.

My Natural Programming Project is working on making software development easier to learn, more effective, and less error prone. An important focus over the last few years has been to discover what are the hard-to-answer questions that developers ask while they are trying to comprehend their programs, and then to develop tools to help answer those questions. For example, when studying programmers working on everyday bugs, we found that they continuously ask “Why” and “Why Not” questions as they try to comprehend what happened. We developed the “Whyline” debugging tool, which allows programmers to directly ask these questions of their programs and get a visualization of the answers. In a small lab study, Whyline increased productivity by a factor of about two. We studied professional programmers trying to understand unfamiliar code and identified over 100 questions they identified as hard-to-answer. In particular, we saw that programmers frequently had specific questions about the feasible execution paths, so we developed a new visualization tool to directly present this information. When trying to use unfamiliar APIs, such as the Java SDK and the SAP eSOA APIs, we discovered some common patterns that make programmers up to 10 times slower in finding and understanding how to use the appropriate methods, so we developed new tools to assist them. This talk will provide an overview of our studies and resulting tools that address program comprehension issues.

Structure makes data more useful, but also makes data entry more cumbersome. Studies have found that this is especially true on mobile devices, as mobile users often reject structured personal information management tools because the structure is too restrictive and makes entering data slower. To overcome these problems, we introduce a new data entry technique that lets users create customized structured data in an unstructured manner. We use a novel notepad-like editing interface with built-in data detectors that allow users to specify structured data implicitly and reuse the structures when desired. To minimize the amount of typing, it provides intelligent, context-sensitive autocomplete suggestions using personal and public databases that contain candidate information to be entered. We implemented these mechanisms in an example application called Listpad. Our evaluation shows that people using Listpad create customized structured data 16% faster than using a conventional mobile database tool. The speed further increases to 42% when the fields can be autocompleted.

A key use for mobile devices is to search and view online information while on the go. As a result, many mobile applications serve as front ends for online databases. While there are many thousands of data sources that provide web service APIs giving access to their databases, creating mobile applications to use those sources requires significant mobile programming knowledge and a significant amount of time. We introduce Spinel, a plug-in architecture for Android, and a set of web-based configuration tools that together enable users to connect mobile applications to new data sources without programming. Spinel also provides APIs that make it easy for developers to create new applications that use those data sources. We provide three demonstration Android applications that use such data: Listpad for entering personal lists, Listviewer for viewing results of data queries, and Mapviewer for displaying query results on a map. An informal usability study showed that users could successfully attach new data sources to those applications.

Conventional version control systems save code changes at each check-in. Recently, some development environments retain more fine-grain changes. However, providing tools for developers to use those histories is not a trivial task, due to the difficulties in visualizing the history. We present two visualizations of fine-grained code change history, which actively interact with the code editor: a timeline visualization, and a code history diff view. Our timeline and filtering options allow developers to navigate through the history and easily focus on the information they need. The code history diff view shows the history of any particular code fragment, allowing developers to move through the history simply by dragging the marker back and forth through the timeline to instantly see the code that was in the snippet at any point in the past. We augment the usefulness of these visualizations with richer editor commands including selective undo and search, which are all implemented in an Eclipse plug-in called “Azurite”. Azurite helps developers with answering common questions developers ask about the code change history that have been identified by prior research. In addition, many of users' backtracking tasks can be achieved using Azurite, which would be tedious or error-prone otherwise.

Examples have been widely used in the area of web design to help web authors create web pages. However, without actually understanding how an example is constructed, people often have trouble extracting the elements they want and incorporating them into their own design. This paper introduces WebCrystal, a web development tool that helps users understand how a web page is built. WebCrystal contributes novel interaction techniques that let the user quickly access HTML and CSS information by selecting questions regarding how a selected element is designed. It provides answers using a textual description and a customized code snippet that can be copied-and-pasted to recreate the desired properties. WebCrystal also supports combining the styles and structures from multiple elements into the generated code snippet, and provides visualizations on the web page itself to explain layout relationships. Our user study shows that WebCrystal helped both novice and experienced developers complete more tasks successfully using significantly less time.

Code completion menus have replaced standalone API browsers for most developers because they are more tightly integrated into the development workflow. Refinements to the code completion menu that incorporate additional sources of information have similarly been shown to be valuable, even relative to standalone counterparts offering similar functionality. In this paper, we describe active code completion, an architecture that allows library developers to introduce interactive and highly-specialized code generation interfaces, called palettes, directly into the editor. Using several empirical methods, we examine the contexts in which such a system could be useful, describe the design constraints governing the system architecture as well as particular code completion interfaces, and design one such system, named Graphite, for the Eclipse Java development environment. Using Graphite, we implement a palette for writing regular expressions as our primary example and conduct a small pilot study. In addition to showing the feasibility of this approach, it provides further evidence in support of the claim that integrating specialized code completion interfaces directly into the editor is valuable to professional developers.

Interactive behaviors in GUIs are often described in terms of states, transitions, and constraints, where the constraints only hold in certain states. These constraints maintain relationships among objects, control the graphical layout, and link the user interface to an underlying data model. However, no existing Web implementation technology provides direct support for all of these, so the code for maintaining constraints and tracking state may end up spread across multiple languages and libraries. In this paper we describe ConstraintJS, a system that integrates constraints and finite-state machines (FSMs) with Web languages. A key role for the FSMs is to enable and disable constraints based on the interface's current mode, making it possible to write constraints that sometimes hold. We illustrate that constraints combined with FSMs can be a clearer way of defining many interactive behaviors with a series of examples.

Developers navigate and reason about call graphs throughout investigation and debugging activities. This is often difficult: developers can spend tens of minutes answering a single question, get lost and disoriented, and erroneously make assumptions, causing bugs. To address these problems, we designed a new form of interactive call graph visualization - REACHER. Instead of leaving developers to manually traverse the call graph, REACHER lets developers search along control flow. The interactive call graph visualization encodes a number of properties that help developers answer questions about causality, ordering, type membership, repetition, choice, and other relationships. And developers remain oriented while navigating. To evaluate REACHER'S benefits, we conducted a lab study in which 12 participants answered control flow questions. Compared to an existing IDE, participants with REACHER were over 5 times more successful in significantly less time. All enthusiastically preferred REACHER, with many positive comments.

Summary form only given. Above showsthat the next generation of mobile applications for MOPs need to be part of a much large mobile information system. Such system should be capable of receiving information from a large number of users, intelligently aggregating the information to useful, meaningful, real-time information and making them available to users. Users should be able to make queries and quickly find information that would enhance their day to day activities. It should also have the functionalities to detect different and unusual situations, provide alerts to users in unusual circumstances and emergencies so that users can respond accordingly. These next generation mobile applications for MOPs will open up a very large market currently estimated to be around 3 billion people [13]. These applications will empower them to be information users as well as information producers. Such applications would result in up lifting of economic conditions of the MOPs and transformation of social processes.

In this paper, we propose a complementary technique called active code completion. When the developer invokes the code completion menu, the editor looks for a palette definition associated with the type of the expression being entered. If found, an option to use this palette is added to the code completion menu. When the developer selects this option, source code is not inserted immediately. Instead, the palette definition takes control of the code completion interface. The developer can then interact with this interface to provide parameters and other information related to her intent, and receive immediate feedback about the effect these choices will have on the object's behavior. When the developer indicates that she is satisfied with these choices, the palette generates code that is inserted at the cursor.

We present Apatite, a new tool that aids users in learning and understanding a complex API by visualizing the common associations between its various components. Current object-oriented API documentation is usually navigated in a fixed tree structure, starting with a package and then filtering by a specific class. For large APIs, this scheme is overly restrictive, because it prevents users from locating a particular action without first knowing which class it belongs to. Apatite's design instead enables users to search across any level of an API's hierarchy. This is made possible by the introduction of a novel interaction technique that presents popular items from multiple categories simultaneously, determining their relevance by approximating the strength of their association using search engine data. The design of Apatite was refined through iterative usability testing, and it has been released publicly as a web application.

When designing novel GUI controls, interaction designers are challenged by the "immaterial" materiality of the digital domain; they lack tools that effectively support a reflecting conversation with the material of software as they attempt to conceive, refine, and communicate their ideas. To investigate this situation, we conducted two participatory design workshops. In the first workshop, focused on conceiving, we observed that designers want to invent controls by exploring gestures, context, and examples. In the second workshop, on refining and communicating, designers proposed tools that could refine movement, document context through usage scenarios, and support the use of examples. In this workshop they struggled to effectively communicate their ideas for developers because their ideas had not been fully explored. In reflecting on this struggle, we began to see an opportunity for the output of a design tool to be a boundary object that would allow for an ongoing conversation between the design and the material of software, in which the developer acts as a mediator for software.

A reachability question is a search across feasible paths through a program for target statements matching search criteria. In three separate studies, we found that reachability questions are common and often time consuming to answer. In the first study, we observed 13 developers in the lab and found that half of the bugs developers inserted were associated with reachability questions. In the second study, 460 professional software developers reported asking questions that may be answered using reachability questions more than 9 times a day, and 82% rated one or more as at least somewhat hard to answer. In the third study, we observed 17 developers in the field and found that 9 of the 10 longest activities were associated with reachability questions. These findings suggest that answering reachability questions is an important source of difficulty understanding large, complex codebases.

RADAR is a multiagent system with a mixed-initiative user interface designed to help office workers cope with email overload. RADAR agents observe experts to learn models of their strategies and then use the models to assist other people who are working on similar tasks. The agents' assistance helps a person to transition from the normal email-centric workflow to a more efficient task-centric workflow. The Email Classifier learns to identify tasks contained within emails and then inspects new emails for similar tasks. A novel task-management user interface displays the found tasks in a to-do list, which has integrated support for performing the tasks. The Multitask Coordination Assistant learns a model of the order in which experts perform tasks and then suggests a schedule to other people who are working on similar tasks. A novel Progress Bar displays the suggested schedule of incomplete tasks as well as the completed tasks. A large evaluation demonstrated that novice users confronted with an email overload test performed significantly better (a 37% better overall score with a factor of four fewer errors) when assisted by the RADAR agents.

In the past decade there has been spectacular growth in the number and size of third-party libraries, frameworks, toolkits and other Application Programming Interfaces (APIs) available to modern software developers. However, the time-saving advantages of code re-use are commonly hampered by the difficulty in finding the correct methods for a given task among the thousands of irrelevant ones. We have developed a tool called Apatite that helps address this issue by letting programmers browse APIs by viewing associations between their components. Apatite indicates which items of an API are popular in different contexts and allows browsing by initially selecting verbs (methods and actions) in addition to classes and packages. The associations are calculated by leveraging existing search engine data and source code, and verbs are identified by parsing the documentation descriptions. Apatite is available on the web and is being used by developers worldwide on a regular basis.

Calcite is a new Eclipse plugin that helps address the difficulty of understanding and correctly using an API. Calcite finds the most popular ways to instantiate a given class or interface by using code examples. To allow the users to easily add these object instantiations to their code, Calcite adds items to the popup completion menu that will insert the appropriate code into the user's program. Calcite also uses crowd sourcing to add to the menu instructions in the form of comments that help the user perform functions that people have identified as missing from the API. In a user study, Calcite improved users' success rate by 40%.

When software developers want to understand the reason for a program's behavior, they must translate their questions about the behavior into a series of questions about code, speculating about the causes in the process. The Whyline is a new kind of debugging tool that avoids such speculation by instead enabling developers to select a question about program output from a set of “why did and why didn't” questions extracted from the program's code and execution. The tool then finds one or more possible explanations for the output in question. These explanations are derived using a static and dynamic slicing, precise call graphs, reachability analyses, and new algorithms for determining potential sources of values. Evaluations of the tool on two debugging tasks showed that developers with the Whyline were three times more successful and twice as fast at debugging, compared to developers with traditional breakpoint debuggers. The tool has the potential to simplify debugging and program understanding in many software development contexts.

Debugging and diagnostic tools are some of the most important software development tools, but most expect developers choose the right code to inspect. Unfortunately, this rarely occurs. A new tool called the Whyline is described which avoids such speculation by allowing developers to select questions about a program's output. The tool then helps developers work backwards from output to its causes. The prototype, which supports Java programs, was evaluated in an experiment in which participants investigated two real bug reports from an open source project using either the Whyline or a breakpoint debugger. Whyline users were successful about three times as often and about twice as fast compared to the control group, and were extremely positive about the tool's ability to simplify diagnostic tasks in software development work.

When users combine data from multiple sources into a spreadsheet or dataset, the result is often a mishmash of different formats, since phone numbers, dates, course numbers and other string-like kinds of data can each be written in many different formats. Although spreadsheets provide features for reformatting numbers and a few specific kinds of string data, they do not provide any support for the wide range of other kinds of string data encountered by users. We describe a user interface where a user can describe the formats of each kind of data. We provide an algorithm that uses these formats to automatically generate reformatting rules that transform strings from one format to another. In effect, our system enables users to create a small expert system called a "tope" that can recognize and reformat instances of one kind of data. Later, as the user is working with a spreadsheet, our system recommends appropriate topes for validating and reformatting the data. With a recall of over 80% for a query time of under 1 second, this algorithm is accurate enough and fast enough to make useful recommendations in an interactive setting. A laboratory experiment shows that compared to manual typing, users can reformat sample spreadsheet data more than twice as fast by creating and using topes.

For developers debugging their own code, augmenting the code of others, or trying to learn the implementation details of interactive behaviors, understanding how web pages work is a fundamental problem. FireCrystal is a new Firefox extension that allows developers to indicate interactive behaviors of interest, and shows the specific code (Javascript, CSS, and HTML) that is responsible for those behaviors. FireCrystal provides an execution timeline that users can scrub back and forth, and the ability to select items of interest in the actual web page UI to see the associated code. FireCrystal may be especially useful for developers who are trying to learn the implementation details of interactive behaviors, so they can reuse these behaviors in their own web site.

Repositories of code written by end-user programmers are beginning to emerge, but when a piece of code is new or nobody has yet reused it, then current repositories provide users with no information about whether that code might be appropriate for reuse. Addressing this problem requires predicting reusability based on information that exists when a script is created. To provide such a model for web macro scripts, we identified script traits that might plausibly predict reuse, then used IBM CoScripter repository logs to statistically test how well each corresponded to reuse. We then built a machine learning model that combines the useful traits and evaluated how well it can predict four different types of reuse that we saw in the repository logs. Our model was able to predict reuse from a surprisingly small set of traits. It is simple enough to be explained in only 6-11 rules, making it potentially viable for integration in repository search engines for end-user programmers.

Jadeite is a new Javadoc-like API documentation system that takes advantage of multiple users' aggregate experience to reduce difficulties that programmers have learning new APIs. Previous studies have shown that programmers often guessed that certain classes or methods should exist, and looked for these in the API. Jadeite's ldquoplaceholdersrdquo let users add new ldquopretendrdquo classes or methods that are displayed in the actual API documentation, and can be annotated with the appropriate APIs to use instead. Since studies showed that programmers had difficulty finding the right classes from long lists in documentation, Jadeite takes advantage of usage statistics to display commonly used classes more prominently. Programmers had difficulty discovering how to instantiate objects, so Jadeite uses a large corpus of sample code to automatically the most common ways to construct an instance of any given class. An evaluation showed that programmers were about three times faster at performing common tasks with Jadeite than with standard Javadoc.

Over six years, we iterated on the design of a language for describing the functionality of appliances, such as televisions, telephones, VCRs, and copiers. This language has been used to describe more than thirty diverse appliances, and these descriptions have been used to automatically generate both graphical and speech user interfaces on handheld computers, mobile phones, and desktop computers. In this article, we describe the final design of our language and analyze the key design choices that led to this design. Through this analysis, we hope to provide a useful guide for the designers of future user interface description languages.

Sometimes people cannot remember the names or locations of things on their computer, but they can remember what other things are associated with them. We created Feldspar, the first system that fully supports this associative retrieval of personal information on the computer. Feldspar's contributions include (1) an intuitive user interface that allows users to find information by interactively and incrementally specifying multiple levels of associations as retrieval queries, such as: "find the file from the person who I met at an event in May"; and (2) algorithms for collecting the association information and for providing answers to associative queries in real-time. A user study showed that Feldspar is easy to use, and suggested that it might be faster than conventional browsing and searching for these kinds of retrieval tasks. Feldspar could be an important addition to search and browsing tools.

When software developers want to understand the reason for a program's behavior, they must translate their questions about the behavior into a series of questions about code, speculating about the causes in the process. The Whyline is a new kind of debugging tool that avoids such speculation by instead enabling developers to select a question about program output from a set of why did and why didn't questions derived from the program's code and execution. The tool then finds one or more possible explanations for the output in question, using a combination of static and dynamic slicing, precise call graphs, and new algorithms for determining potential sources of values and explanations for why a line of code was not reached. Evaluations of the tool on one task showed that novice programmers with the Whyline were twice as fast as expert programmers without it. The tool has the potential to simplify debugging in many software development contexts.

Programmers often omit input validation when inputs can appear in many different formats or when validation criteria cannot be precisely specified. To enable validation in these situations, we present a new technique that puts valid inputs into a consistent format and that identifies "questionable" inputs which might be valid or invalid, so that these values can be double-checked by a person or a program. Our technique relies on the concept of a "tope", which is an application-independent abstraction describing how to recognize and transform values in a category of data. We present our definition of topes and describe a development environment that supports the implementation and use of topes. Experiments with web application and spreadsheet data indicate that using our technique improves the accuracy and reusability of validation code and also improves the effectiveness of subsequent data cleaning such as duplicate identification.

End-user programming tools for creating spreadsheets and webforms offer no data types except "string" for storing many kinds of data, such as person names and street addresses. Consequently, these tools cannot automatically validate these data.

To better understand what makes Application Programming Interfaces (APIs) hard to use and how to improve them, recent research has begun studying programmers' strategies and use of APIs. It was found that method placement --- on which class or classes a method is placed --- can have large usability impact in object-oriented APIs. This was because programmers often start their exploration of an API from one "main" object, and were slower finding other objects that were not referenced in the methods of the main object. For example, while mailServer.send(mailMessage) might make sense, if programmers often begin their API explorations from the MailMessage class, then this makes it harder to find the MailServer class than the alternative mailMessage.send(mailServer). This is interesting because many real APIs place methods essential to common objects on other, helper objects. Alternate versions of three different APIs were compared, and it was found that programmers gravitated toward the same starting classes and were dramatically faster --- between 2 to 11 times --- combining multiple objects when a method on the starting class referred to the other class.

An important part of many programming tasks is the use of libraries and other forms of Application Programming Interfaces (APIs). Programming via web services using a Service-Oriented Architecture (SOA) is an emerging form of API usage. Web services in a business context (called enterprise SOA or E-SOA) add additional complexity in terms of the number of the services, the variety of internal data structures, and service interdependencies. After altering existing Human-Computer Interaction (HC[) methodologies to address the unique context of software development for SOA, we evaluated a large E-SOA API and identified many usability challenges. Prominent results include difficulties developers encountered while assembling data structures in web service parameters, cycles of errors due to unclear control parameters within data structures, and difficulties with understanding long identifier names. We recommend a tolerance for unspecified objects in automatically-generated web service proxy code, consistent data structures in parame ters across services, and encoding optional namespace schemes into WSDL files.

Designers are skilled at sketching and prototyping the look of interfaces, but to explore various behaviors (what the interface does in response to input) typically requires programming using Javascript, ActionScript for Flash, or other languages. In our survey of 259 designers, 86% reported that the behavior is more difficult to prototype than the appearance. Often (78% of the time), designing the behavior requires collaborating with developers, but 76% of designers reported that communicatin1g the behavior to developers was more difficult than the appearance. Other results include that annotations such as arrows and paragraphs of text are used on top of sketches and storyboards to explain behaviors, and designers want to explore multiple versions of behaviors, but today’s tools make this difficult. The results provide new ideas for future tools.

While a designer's focus used to be the design of non-interactive elements such as graphics or animations, today's designers deal with various levels of interactivity such as mouse, keyboard and touch screen interaction. Unfortunately, it is challenging for designers to create these diverse interactions since most implementation tools such as Flash require the use of conventional programming languages and do not support the natural expressions used by designers. To better understand how designers think about interactive behaviors, we conducted a lab study where designers and programmers described various primitive and composite interactive behaviors using their own language. From this, we learned that there is significant commonality among designers in terms of the verbs, syntax, and structure when describing interactivity. These results can help guide the way to building more natural programming languages and environments for designers to facilitate the development of interactive behaviors.

We conducted a user study that demonstrates that automatically generated interfaces can support better usability through increased flexibility in two dimensions. First, we show that automatic generation can improve usability by moving interfaces that are constrained by cost and poor interaction primitives to another device with better interactive capabilities: subjects were twice as fast and four times as successful at completing tasks with automatically generated interfaces on a PocketPC device as with the actual appliance interfaces. Second, we show that an automatic generator can improve usability by automatically ensuring that new interfaces are generated to be consistent with users' previous experience: subjects were also twice as fast using interfaces consistent with their experiences as compared to normally generated interfaces. These two results demonstrate that automatic interface generation is now viable and especially desirable where users will benefit from individualized interfaces or where human designers are constrained by cost and other factors.

A gestural text entry method for mobile is presented. Unlike most mobile phone text entry methods, which rely on repeatedly pressing buttons, our gestural method uses an isometric joystick and the EdgeWrite alphabet to allow users to write by making letter-like "pressure strokes." In a 15-session study comparing character-level EdgeWrite to Multitap, subjects' speeds were statistically indistinguishable, reaching about 10 WPM. In a second 15-session study comparing word-level EdgeWrite to T9, the same subjects were again statistically indistinguishable, reaching about 16 WPM. Uncorrected errors were low, around 1% or less for each method. In addition, subjective results favored EdgeWrite. Overall, results indicate that our isometric joystick-based method is highly competitive with two commercial keypad-based methods, opening the way for keypad-less designs and text entry on tiny devices. Additional results showed that a joystick on the back could be used at about 70% of the speed of the front, and the front joystick could be used eyes-free at about 80% of the speed of normal use.

The increasing quantity and complexity of in-vehicle systems creates a demand for user interfaces which are suited to driving. The steering wheel is a common location for the placement of buttons to control navigation, entertainment, and environmental systems, but what about a small touchpad? To investigate this question, we embedded a Synaptics StampPad in a computer game steering wheel and evaluated seven methods for selecting from a list of over 3000 street names. Selection speed was measured while stationary and while driving a simulator. Results show that the EdgeWrite gestural text entry method is about 20% to 50% faster than selection-based text entry or direct list-selection methods. They also show that methods with slower selection speeds generally resulted in faster driving speeds. However, with EdgeWrite, participants were able to maintain their speed and avoid incidents while selecting and driving at the same time. Although an obvious choice for constrained input, on-screen keyboards generally performed quite poorly.

The usability of software APIs is an important and infrequently researched topic. A user study comparing the usability of the factory pattern and constructors in API designs found highly significant results indicating that factories are detrimental to API usability in several varied situations. The results showed that users require significantly more time (p = 0.005) to construct an object with a factory than with a constructor while performing both context-sensitive and context- free tasks. These results suggest that the use of factories can and should be avoided in many cases where other techniques, such as constructors or class clusters, can be used instead.

Little is known about how developers think about design during code modification tasks or how experienced developers' design knowledge helps them work more effectively. We performed a lab study in which thirteen developers worked for 3 hours under-standing the design of a 54 KLOC open source application. Par-ticipants had from 0 to 10.5 years of industry experience and were grouped into three "experts" and ten "novices." We observed that participants spent their time seeking, learning, critiquing, explain-ing, proposing, and implementing facts about the code such as "getFoldLevel has effects". These facts served numerous roles, such as suggesting changes, constraining changes, and predicting the amount of additional investigation necessary to make a change. Differences between experts and novices included that the experts explained the root cause of the design problem and made changes to address it, while novice changes addressed only the symptoms. Experts did not read more methods but also did not visit some methods novices wasted time understanding. Experts talked about code in terms of abstractions such as "caching" while novices more often described code statement by statement. Ex-perts were able to implement a change faster than novices. Experts perceived problems novices did not and were able to explain facts novices could not. These findings have interesting implications for future tools.

Web macros automate interactions with Web sites and related information systems. Though Web macro recorders and players have grown in sophistication over the past decade, these tools cannot yet meet many needs of users in daily life. Based on observations of browser users, we have compiled ten scenarios describing tasks that users would benefit from automating. Our analysis of these scenarios yields specific requirements that Web macro tools should support if those tools are to be applicable to these real-life tasks. Our set of requirements constitutes a benchmark for evaluating tools.

When creating new application programming interfaces (APIs), designers must make many decisions. These decisions affect the quality of the resulting APIs in terms of performance (such as speed and memory usage), power (expressiveness, extensibility and evolvability) and usability (learnability, productivity and error prevention). Experienced API designers have written recommendations on how to design APIs, offering their opinions on various API design decisions. Additionally, empirical research has begun to measure the usability tradeoffs of specific design decisions. While previous work has offered specific suggestions, there has not been a clear description of the design space of all possible API design decisions, or the quality attributes that these decisions affect. This paper puts existing API design recommendations into context by mapping out the space of API design decisions and API quality attributes.

introduction Share on Introduction to special issue on computers and accessibility Authors: Andrew Sears UMBC, Baltimore, MD UMBC, Baltimore, MDView Profile , Vicki L. Hanson IBM T. J. Watson Research Center, Hawthrone, NY IBM T. J. Watson Research Center, Hawthrone, NYView Profile , Brad Myers Carnegie Mellon University, Pittsburgh, PA Carnegie Mellon University, Pittsburgh, PAView Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 14Issue 3September 2007 pp 11–eshttps://doi.org/10.1145/1279700.1279701Published:01 September 2007Publication History 1citation810DownloadsMetricsTotal Citations1Total Downloads810Last 12 Months5Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my Alerts New Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

We present a major extension to our previous work on Trackball EdgeWrite--a unistroke text entry method for trackballs--by taking it from a character-level technique to a word-level one. Our design is called stroke-based word completion, and it enables efficient word selection as part of the stroke-making process. Unlike most word completion designs, which require users to select words from a list, our technique allows users to select words by performing a fluid crossing gesture. Our theoretical model shows this word-level design to be 45.0% faster than our prior model for character-only strokes. A study with a subject with spinal cord injury comparing Trackball EdgeWrite to the onscreen keyboard WiViK, both using word prediction and completion, shows that Trackball EdgeWrite is competitive with WiViK in speed (12.09 vs. 11.82 WPM) and accuracy (3.95% vs. 2.21% total errors), but less visually tedious and ultimately preferred. The results also show that word-level Trackball EdgeWrite is 46.5% faster and 36.7% more accurate than our subject's prior peak performance with character-level Trackball EdgeWrite, and 75.2% faster and 40.2% more accurate than his prior peak performance with his preferred on-screen keyboard. An additional evaluation of the same subject over a two-month field deployment shows a 43.9% reduction in unistrokes due to strokebased word completion in Trackball EdgeWrite.

Recent advances in programming environments have focused on improving programmer productivity by utilizing the inherent structure in computer programs. However, because these environments represent code as plain text, it is difficult and sometimes impossible to embed interactive tools, annotations, and alternative views in the code itself. Barista is an implementation framework that enables the creation of such user interfaces by simplifying the implementation of editors that represent code internally as an abstract syntax tree and maintain a corresponding, fully structured visual representation on-screen. Barista also provides designers of editors with a standard text-editing interaction technique that closely mimics that of conventional text editors, overcoming a central usability issue of previous structured code editors.

Modern applications such as Microsoft Word have many automatic features and hidden dependencies that are frequently helpful but can be mysterious to both novice and expert users. The ""Crystal"" application framework provides an architecture and interaction techniques that allow programmers to create applications that let the user ask a wide variety of questions about why things did and did not happen, and how to use the related features of the application without using natural language. A user can point to an object or a blank space and get a popup list of questions about it, or the user can ask about recent actions from a temporal list. Parts of a text editor were implemented to show that these techniques are feasible, and a user test suggests that they are helpful and well-liked.

A problem with many of today's appliance interfaces is that they are inconsistent. For example, the procedure for setting the time on alarm clocks and VCRs differs, even among different models made by the same manufacturer. Finding particular functions can also be a challenge, because appliances often organize their features differently. This paper presents a system, called Uniform, which approaches this problem by automatically generating remote control interfaces that take into account previous interfaces that the user has seen during the generation process. Uniform is able to automatically identify similarities between different devices and users may specify additional similarities. The similarity information allows the interface generator to use the same type of controls for similar functions, place similar functions so that they can be found with the same navigation steps, and create interfaces that have a similar visual appearance.

We present a new gestural text entry method for trackballs. The method uses the mouse cursor and relies on crossing instead of pointing. A user writes in fluid Roman-like unistrokes by ""pulsing"" the trackball in desired letter patterns. We examine this method both theoretically using the Steering Law and empirically in two studies. Our studies show that able-bodied users who were unfamiliar with trackballs could write at about 10 wpm with <4% total errors after 45 minutes. In eight sessions, a motor-impaired trackball user peaked at 7.11 wpm with 0% uncorrected errors, compared to 5.95 wpm with 0% uncorrected errors with an on-screen keyboard. Over sessions, his speeds were significantly faster with our gestural method than with an on-screen keyboard. A former 15-year veteran of on-screen keyboards, he now uses our gestural method instead.

We present a new 4-key text entry method that, unlike most few-key methods, is gestural instead of selection-based. Importantly, its gestures mimic the writing of Roman letters for high learnability. We compare this new 4-key method to predominant 3-key and 5-key methods theoretically using KSPC and empirically using a longitudinal study of 5 subjects over 10 sessions. The study includes an evaluation of the 4-key method without any on-screen visualization-an impossible condition for the selection-based methods. Our results show that the new 4-key method is quickly learned, becoming faster than the 3-key and 5-key methods after just ~10 minutes of writing, although it produces more errors. Interestingly, removing a visualization of the gestures being made causes no detriment to the 4-key method, which is an advantage for eyes-free text entry.

Systems of connected appliances, such as home theaters and presentation rooms, are becoming commonplace in our homes and workplaces. These systems are often difficult to use, in part because users must determine how to split the tasks they wish to perform into sub-tasks for each appliance and then find the particular functions of each appliance to complete their sub-tasks. This paper describes Huddle, a new system that automatically generates task-based interfaces for a system of multiple appliances based on models of the content flow within the multi-appliance system.

We present the design and implementation of a word-level stroking system called Fisch, which is intended to improve the speed of character-level unistrokes. Importantly, Fisch does not alter the way in which character-level unistrokes are made, but allows users to gradually ramp up to word-level unistrokes by extending their letters in minimal ways. Fisch relies on in-stroke word completion, a flexible design for fluidly turning unistroke letters into whole words. Fisch can be memorized at the motor level since word completions always appear at the same positions relative to the strokes being made. Our design for Fisch is suitable for use with any unistroke alphabet. We have implemented Fisch for multiple versions of EdgeWrite, and results show that Fisch reduces the number of strokes during entry by 43.9% while increasing the rate of entry. An informal test of "record speed" with the stylus version resulted in 50-60 wpm with no uncorrected errors.

There is little understanding of how people describe software problems, but a variety of tools solicit, manage, and analyze these descriptions in order to streamline software development. To inform the design of these tools and generate ideas for new ones, an study of nearly 200,000 bug report titles was performed. The titles of the reports generally described a software entity or behavior, its inadequacy, and an execution context, suggesting new designs for more structured report forms. About 95% of noun phrases referred to visible software entities, physical devices, or user actions, suggesting the feasibility of allowing users to select these entities in debuggers and other tools. Also, the structure of the titles exhibited sufficient regularity to parse with an accuracy of 89%, enabling a number of new automated analyses. These findings and others have many implications for tool design and software engineering

Information workers such as administrative staff, consultants, and their managers constitute one of the largest groups of end users, yet little research about their usage of programming features is available to guide development of end user programming tools. In this paper, we describe our survey of over 800 information workers and our analysis of their feature usage in applications such as spreadsheets, browsers, and databases. Our factor analysis reveals three clusters of features - macro features, linked structure features, and imperative features - such that information workers with an inclination to use a feature in each cluster also were inclined to use other features in that cluster, even though each cluster spans several tools. We discuss the implications for research aimed at providing end user programming tools for information workers

Because software libraries are numerous and large, learning how to use them is a common and problematic task for experienced programmers and novices alike. Internet search engines such as Google have emerged as important resources to help programmers successfully use APIs. However, observations of programmers using Web search have revealed problems and inefficiencies in their use. We present a new prototype search tool called Mica that augments standard Web search results to help programmers find the right API classes and methods given a description of the desired functionality, and help programmers find examples when they already know which methods to use. Mica works by using the Google Web APIs to find relevant pages, and then analyzing the content of those pages to extract the most relevant programming terms and to classify the type of each result

Creativity support tools is a research topic with high risk but potentially very high payoff. The goal is to develop improved software and user interfaces that empower users to be not only more productive but also more innovative. Potential users include software and other engineers, diverse scientists, product and graphic designers, architects, educators, students, and many others. Enhanced interfaces could enable more effective searching of intellectual resources, improved collaboration among teams, and more rapid discovery processes. These advanced interfaces should also provide potent support in hypothesis formation, speedier evaluation of alternatives, improved understanding through visualization, and better dissemination of results. For creative endeavors that require composition of novel artifacts (e.g., computer programs, scientific papers, engineering diagrams, symphonies, artwork), enhanced interfaces could facilitate exploration of alternatives, prevent unproductive choices, and enable easy backtracking. This U.S. National Science Foundation sponsored workshop brought together 25 research leaders and graduate students to share experiences, identify opportunities, and formulate research challenges. Two key outcomes emerged: (a) encouragement to evaluate creativity support tools through multidimensional in-depth longitudinal case studies and (b) formulation of 12 principles for design of creativity support tools.

Recent improvements in text entry error rate measurement have enabled the running of text entry experiments in which subjects are free to correct errors (or not) as they transcribe a presented string. In these “unconstrained” experiments, it is no longer necessary to force subjects to unnaturally maintain synchronicity with presented text for the sake of performing overall error rate calculations. However, the calculation of character-level error rates, which can be trivial in artificially constrained evaluations, is far more complicated in unconstrained text entry evaluations because it is difficult to infer a subject's intention at every character. For this reason, prior character-level error analyses for unconstrained experiments have only compared presented and transcribed strings, not input streams. But input streams are rich sources of character-level error information, since they contain all of the text entered (and erased) by a subject. The current work presents an algorithm for the automated analysis of character-level errors in input streams for unconstrained text entry evaluations. It also presents new character-level metrics that can aid method designers in refining text entry methods. To exercise these metrics, we perform two analyses on data from an actual text entry experiment. One analysis, available from the prior work, uses only presented and transcribed strings. The other analysis uses input streams, as described in the current work. The results confirm that input stream error analysis yields richer information for the same empirical data. To facilitate the use of these new analyses, we offer pseudocode and downloadable software for performing unconstrained text entry experiments and analyzing data.

Much of software developers' time is spent understanding unfamiliar code. To better understand how developers gain this understanding and how software development environments might be involved, a study was performed in which developers were given an unfamiliar program and asked to work on two debugging tasks and three enhancement tasks for 70 minutes. The study found that developers interleaved three activities. They began by searching for relevant code both manually and using search tools; however, they based their searches on limited and misrepresentative cues in the code, environment, and executing program, often leading to failed searches. When developers found relevant code, they followed its incoming and outgoing dependencies, often returning to it and navigating its other dependencies; while doing so, however, Eclipse's navigational tools caused significant overhead. Developers collected code and other information that they believed would be necessary to edit, duplicate, or otherwise refer to later by encoding it in the interactive state of Eclipse's package explorer, file tabs, and scroll bars. However, developers lost track of relevant code as these interfaces were used for other tasks, and developers were forced to find it again. These issues caused developers to spend, on average, 35 percent of their time performing the mechanics of navigation within and between source files. These observations suggest a new model of program understanding grounded in theories of information foraging and suggest ideas for tools that help developers seek, relate, and collect information in a more effective and explicit manner.

We present various adaptations of the EdgeWrite unistroke text entry method that work on multiple computer input devices: styluses, touchpads, displacement and isometric joysticks, four keys or buttons, and trackballs. We argue that consistent, flexible, multi-device input is important to both accessibility and to ubiquitous computing. For accessibility, multi-device input means users can switch among devices, distributing strain and fatigue among different muscle groups. For ubiquity, it means users can "learn once, write anywhere," even as new devices emerge. By considering the accessibility and ubiquity of input techniques, we can design for both motor-impaired users and "situationally impaired" able-bodied users who are on-the-go. We discuss the requirements for such input and the challenges of multi-device text entry, such as solving the segmentation problem. This paper accompanies a demonstration of EdgeWrite on multiple devices.

Recently, several innovative tools have found their way into mainstream use in modern development environments. However, most of these tools have focused on creating and modifying code, despite evidence that most of programmers' time is spent understanding code as part of maintenance tasks. If new tools were designed to directly support these maintenance tasks, what types would be most helpful? To find out, a study of expert Java programmers using Eclipse was performed. The study suggests that maintenance work consists of three activities: (1) forming a working set of task-relevant code fragments; (2) navigating the dependencies within this working set; and (3) repairing or creating the necessary code. The study identified several trends in these activities, as well as many opportunities for new tools that could save programmers up to 35% of the time they currently spend on maintenance tasks.

Direct-manipulation editors for structured data are increasingly common. While such editors can greatly simplify the creation of structured data, there are few tools to simplify the creation of the editors themselves. This paper presents Citrus, a new programming language and user interface toolkit designed for this purpose. Citrus offers language-level support for constraints, restrictions and change notifications on primitive and aggregate data, mechanisms for automatically creating, removing, and reusing views as data changes, a library of widgets, layouts and behaviors for defining interactive views, and two comprehensive interactive editors as an interface to the language and toolkit itself. Together, these features support the creation of editors for a large class of data and code.

There are many common spreadsheet errors that traditional spreadsheet systems do not help users find. This paper presents a statically-typed spreadsheet language that adds additional information about the objects that the spreadsheet values represent. By annotating values with both units and labels, users denote both the system of measurement in which the values are expressed as well as the properties of the objects to which the values refer. This information is used during computation to detect some invalid computations and allow users to identify properties of the resulting values.

In 1995, Boehm predicted that by 2005, there would be "55 million performers" of "end user programming" in the United States. The original context and method which generated this number had two weaknesses, both of which we address. First, it relies on undocumented, judgment-based factors to estimate the number of end user programmers based on the total number of end users; we address this weakness by identifying specific end user sub-populations and then estimating their sizes. Second, Boehm's estimate relies on additional undocumented, judgment-based factors to adjust for rising computer usage rates; we address this weakness by integrating fresh Bureau of Labor Statistics (BLS) data and projections as well as a richer estimation method. With these improvements to Boehm's method, we estimate that in 2012 there will be 90 million end users in American workplaces. Of these, we anticipate that over 55 million will use spreadsheets or databases (and therefore may potentially program), while over 13 million will describe themselves as programmers, compared to BLS projections of fewer than 3 million professional programmers. We have validated our improved method by generating estimates for 2001 and 2003, then verifying that our estimates are consistent with existing estimates from other sources.

Power wheelchair joysticks have be used to control a mouse cursor on desktop computers, but they offer no integrated text entry solution, confining users to point-and-click or point-and-dwell with on-screen keyboards. On-screen keyboards reduce useful screen real-estate, exacerbating the need for frequent window management, and impose a secondary focus of attention. By contrast, we present two integrated gestural text entry methods designed for use from power wheelchairs: one for use with joysticks and the other for use with touchpads. Both techniques are adaptations of EdgeWrite, originally a stylus-based unistroke method designed for people with tremor. In a preliminary text entry study of 7 power wheelchair users, we found that EdgeWrite with a touchpad was faster than the on-screen keyboard WiViK with a joystick, and EdgeWrite with a joystick was only slightly slower. These results warranted a multi-session comparison of text entry with EdgeWrite and WiViK using joysticks and touchpads, in which we found touchpads faster than joysticks, and EdgeWrite faster than WiViK with both devices after initial learning periods.

This article provides an overview of the capabilities that we are developing as part of the Pebbles research project for wireless handheld devices such as mobile phones and palm-size computers like Palm Organizers and PocketPCs. Instead of just being used as a phone or organizer, handheld devices can also be used as remote controls for computers and household and office appliances.

One-way constraints have been incorporated in many graphical user interface toolkits because they are simple to learn, easy to write, and can express many types of useful graphical relationships. This paper is an evaluative paper that examines users' experience with one-way constraints in two user interface development toolkits, Garnet and Amulet, over a 15-year time span. The lessons gained from this examination can help guide the design of future constraint systems. The most important lessons are that (1) constraints should be allowed to contain arbitrary code that is written in the underlying toolkit language and does not require any annotations, such as parameter declarations, (2) constraints are difficult to debug and better debugging tools are needed, and (3) programmers will readily use one-way constraints to specify the graphical layout of an application, but must be carefully and time-consumingly trained to use them for other purposes. Copyright © 2005 John Wiley & Sons, Ltd.

Power wheelchair joysticks have been used to control a mouse cursor on desktop computers, but they offer no integrated text entry solution, confining users to point-and-click or point-and-dwell with on-screen keyboards. But on-screen keyboards reduce useful screen real-estate, exacerbate the need for frequent window management, and impose a second focus of attention. By contrast, we present two integrated gestural text entry methods designed for use from power wheelchairs: one for joysticks and the other for touchpads. Both techniques are adaptations of EdgeWrite, originally a stylus-based unistroke method designed for people with tremor. In a preliminary study of 7 power wheelchair users, we found that touchpad EdgeWrite was faster than joystick WiVik, and joystick EdgeWrite was only slightly slower after minimal practice. These findings reflect "walk up and use"-ability and warrant further investigation into extended use.

Debugging is still among the most common and costly of programming activities. One reason is that current debugging tools do not directly support the inquisitive nature of the activity. Interrogative Debugging is a new debugging paradigm in which programmers can ask why did and even why didn't questions directly about their program's runtime failures. The Whyline is a prototype Interrogative Debugging interface for the Alice programming environment that visualizes answers in terms of runtime events directly relevant to a programmer's question. Comparisons of identical debugging scenarios from user tests with and without the Whyline showed that the Whyline reduced debugging time by nearly a factor of 8, and helped programmers complete 40% more tasks.

One of the challenges of using mobile devices for ubiquitous re-mote control is the creation of the user interface. If automatically generated designs are used, then they must be close in quality to hand-designed interfaces. Automatically generated interfaces can be dramatically improved if they use standard conventions to which users are accustomed, such as the arrangement of buttons on a telephone dial-pad or the conventional play, stop, and pause icons on a media player. Unfortunately, it can be difficult for a system to determine where to apply design conventions because each appliance may represent its functionality differently. Smart Templates is a technique that uses parameterized templates in the appliance model to specify when such conventions might be automatically applied in the user interface. Our templates easily adapt to existing appliance models and interface generators on different platforms can apply appropriate design conventions using templates.

We present Citrine, a system that extends the widespread copy-and-paste interaction technique with intelligent transformations, making it useful in more situations. Citrine uses text parsing to find the structure in copied text and allows users to paste the structured information, which might have many pieces, in a single paste operation. For example, using Citrine, a user can copy the text of a meeting request and add it to the Outlook calendar with a single paste. In applications such as Excel, users can teach Citrine by example how to copy and paste data by showing it which fields go into which columns, and can use this to copy or paste many items at a time in a user-defined manner. Citrine can be used with a wide variety of applications and types of data and can be easily extended to work with more. It currently includes parsers that recognize contact information, calendar appointments and bibliographic citations. It works with Internet Explorer, Outlook, Excel, Palm Desktop, EndNote and other applications. Citrine is available to download on the internet.

As programming skills increase in demand and utility, the learnability of end-user programming systems is of utmost importance. However, research on learning barriers in programming systems has primarily focused on languages, overlooking potential barriers in the environment and accompanying libraries. To address this, a study of beginning programmers learning Visual Basic.NET was performed. This identified six types of barriers: design, selection, coordination, use, understanding, and information. These barriers inspire a new metaphor of computation, which provides a more learner-centric view of programming system design.

EdgeWrite is a new unistroke text entry method for handheld devices designed to provide high accuracy and stability of motion for people with motor impairments. It is also effective for able-bodied people. An EdgeWrite user enters text by traversing the edges and diagonals of a square hole imposed over the usual text input area. Gesture recognition is accomplished not through pattern recognition but through the sequence of corners that are hit. This means that the full stroke path is unimportant and recognition is highly deterministic, enabling better accuracy than other gestural alphabets such as Graffiti. A study of able-bodied users showed subjects with no prior experience were 18% more accurate during text entry with Edge Write than with Graffiti (p>.05), with no significant difference in speed. A study of 4 subjects with motor impairments revealed that some of them were unable to do Graffiti, but all of them could do Edge Write. Those who could do both methods had dramatically better accuracy with Edge Write.

People with Muscular Dystrophy (MD) and certain other muscular and nervous system disorders lose their gross motor control while retaining fine motor control. The result is that they lose the ability to move their wrists and arms, and therefore their ability to operate a mouse and keyboard. However, they can often still use their fingers to control a pencil or stylus, and thus can use a handheld computer such as a Palm. We have developed software that allows the handheld to substitute for the mouse and keyboard of a PC, and tested it with four people (ages 10, 12, 27 and 53) with MD. The 12-year old had lost the ability to use a mouse and keyboard, but with our software, he was able to use the Palm to access email, the web and computer games. The 27-year-old reported that he found the Palm so much better that he was using it full-time instead of a keyboard and mouse. The other two subjects said that our software was much less tiring than using the conventional input devices, and enabled them to use computers for longer periods. We report the results of these case studies, and the adaptations made to our software for people with disabilities.

It is difficult to interact with computer displays that are across the room. A popular approach is to use laser pointers tracked by a camera, but interaction techniques using laser pointers tend to be imprecise, error-prone, and slow. Although many previous papers discuss laser pointer interaction techniques, none seem to have performed user studies to help inform the design. This paper reports on two studies of laser pointer interactions that answer some of the questions related to interacting with objects using a laser pointer. The first experiment evaluates various parameters of laser pointers. For example, the time to acquire a target is about 1 second, and the jitter due to hand unsteadiness is about ±8 pixels, which can be reduced to about ±2 to ±4 pixels by filtering. We compared 7 different ways to hold various kinds of laser pointers, and found that a laser pointer built into a PalmOS device was the most stable. The second experiment compared 4 different ways to select objects on a large projected display. We found that tapping directly on a wall-size SmartBoard was the fastest and most accurate method, followed by a new interaction technique that copies the area of interest from the big screen to a handheld. Third in speed was the conventional mouse, and the laser pointer came in last, with a time almost twice as long as tapping on the SmartBoard

Programming languages are the way for a person to express a mental plan in a way that the computer can understand. Therefore, it is appropriate to consider properties of people when designing new programming languages. In our research, we are investigating how people think about algorithms, and how programming languages can be made easier to learn and more effective for people to use. By taking human-productivity aspects of programming languages seriously, designers can more effectively match programming language features with human capabilities and problem solving methods. Human factors methods can be used to measure the effects, so unsubstantiated claims can be avoided.This talk will present a quick summary of new and old results in what is known about people and programming, from areas that are sometimes called "empirical studies of programmers" and "psychology of programming." Much is known about what people find difficult, and what syntax and language features are especially tricky and bug-prone. Our new research has discovered how people naturally think about algorithms and data structures, which can help with making programming languages more closely match people's problem solving techniques.

We describe our system which facilitates collaboration using multiple modalities, including speech, handwriting, gestures, gaze tracking, direct manipulation, large projected touch-sensitive displays, laser pointer tracking, regular monitors with a mouse and keyboard, and wireless networked handhelds. Our system allows multiple, geographically dispersed participants to simultaneously and flexibly mix different modalities using the right interface at the right time on one or more machines. We discuss each of the modalities provided, how they were integrated in the system architecture, and how the user interface enabled one or more people to flexibly use one or more devices.

Several industrial and academic research groups are working to simplify the control of appliances and services by creating a truly universal remote control. Unlike the preprogrammed remote controls available today, these new controllers download a specification from the appliance or service and use it to automatically generate a remote control interface. This promises to be a useful approach because the specification can be made detailed enough to generate both speech and graphical interfaces. Unfortunately, generating good user interfaces can be difficult. Based on user studies and prototype implementations, this paper presents a set of requirements that we have found are needed for automatic interface generation systems to create high-quality user interfaces.

Multiple selections, though heavily used in file managers and drawing editors, are virtually nonexistent in text editing. This paper describes how multiple selections can automate repetitive text editing. Selection guessing infers a multiple selection from positive and negative examples provided by the user. The multiple selection can then be used for inserting, deleting, copying, pasting, or other editing commands. Simultaneous editing uses two levels of inference, first inferring a group of records to be edited, then inferring multiple selections with exactly one selection in each record. Both techniques have been evaluated by user studies and shown to be fast and usable for novices. Simultaneous editing required only 1.26 examples per selection in the user study, approaching the ideal of 1-example PBD. Multiple selections bring many benefits, including better user feedback, fast, accurate inference, novel forms of intelligent assistance, and the ability to override system inferences with manual corrections.

The personal universal controller (PUC) is an approach for improving the interfaces to complex appliances by introducing an intermediary graphical or speech interface. A PUC engages in two-way communication with everyday appliances, first downloading a specification of the appliance's functions, and then automatically creating an interface for controlling that appliance. The specification of each appliance includes a high-level description of every function, a hierarchical grouping of those functions, and dependency information, which relates the availability of each function to the appliance's state. Dependency information makes it easier for designers to create specifications and helps the automatic interface generators produce a higher quality result. We describe the architecture that supports the PUC, and the interface generators that use our specification language to build high-quality graphical and speech interfaces.

The proliferation of wireless handheld devices is placing the World Wide Web in the palms of users, but this convenience comes at a high interactive cost. The Web that came of age on the desktop is ill-suited for use on the small displays of handhelds. Today, handheld browsing often feels like browsing on a PC with a shrunken desktop. Overreliance on scrolling is a big problem in current handheld browsing. Users confined to viewing a small portion of each page often lack a sense of the overall context --- they may feel lost in a large page and be forced to remember the locations of items as those items scroll out of view. In this paper, we present a synthesis of interaction techniques to address these problems. We implemented these techniques in a prototype, WebThumb, that can browse the live Web.

When users handle large amounts of data, errors are hard to notice. Outlier finding is a new way to reduce errors by directing the user's attention to inconsistent data which may indicate errors. We have implemented an outlier finder for text, which can detect both unusual matches and unusual mismatches to a text pattern. When integrated into the user interface of a PBD text editor and tested in a user study, outlier finding substantially reduced errors.

Although commercial dictation systems and speech-enabled telephone voice user interfaces have become readily available, speech recognition errors remain a serious problem in the design and implementation of speech user interfaces. Previous work hypothesized that switching modality could speed up interactive correction of recognition errors. This article presents multimodal error correction methods that allow the user to correct recognition errors efficiently without keyboard input. Correction accuracy is maximized by novel recognition algorithms that use context information for recognizing correction input. Multimodal error correction is evaluated in the context of a prototype multimodal dictation system. The study shows that unimodal repair is less accurate than multimodal error correction. On a dictation task, multimodal correction is faster than unimodal correction by respeaking. The study also provides empirical evidence that system-initiated error correction (based on confidence measures) may not expedite error correction. Furthermore, the study suggests that recognition accuracy determines user choice between modalities: while users initially prefer speech, they learn to avoid ineffective correction modalities with experience. To extrapolate results from this user study, the article introduces a performance model of (recognition-based) multimodal interaction that predicts input speed including time needed for error correction. Applied to interactive error correction, the model predicts the impact of improvements in recognition technology on correction speeds, and the influence of recognition accuracy and correction method on the productivity of dictation systems. This model is a first step toward formalizing multimodal interaction.

One-way, dataflow constraints are commonly used in graphical interface toolkits, programming environments, and circuit applications. Previous papers on dataflow constraints have focused on the design and implementation of individual algorithms. In contrast, this article focuses on the lessons we have learned from a decade of implementing competing algorithms in the Garnet and Amulet graphical interface toolkits. These lessons reveal the design and implementation tradeoffs for different one-way, constraint satisfaction algorithms. The most important lessons we have learned are that (1) mark-sweep algorithms are more efficient than topological ordering algorithms; (2) lazy and eager evaluators deliver roughly comparable performance for most applications; and (3) constraint satisfaction algorithms have more than adequate speed, except that the storage required by these algorithms can be problematic.

We performed several experiments using a Personal Digital Assistant (PDA) as an input device in the non-dominant hand along with a mouse in the dominant hand. A PDA is a small hand-held palm-size computer like a 3Com Palm Pilot or a Windows CE device. These are becoming widely available and are easily connected to a PC. Results of our experiments indicate that people can accurately and quickly select among a small numbers of buttons on the PDA using the left hand without looking, and that, as predicted, performance does decrease as the number of buttons increases. Homing times to move both hands between the keyboard and devices are only about 10% to 15% slower than times to move a single hand to the mouse, suggesting that acquiring two devices does not cause a large penalty. In an application task, we found that scrolling web pages using buttons or a scroller on the PDA matched the speed of using a mouse with a conventional scroll bar, and beat the best two-handed times reported in an earlier experiment. These results will help make two-handed interactions with computers more widely available and more effective.

A user interface software tool helps developers design and implement the user interface. Research on past tools has had enormous impact on today's developers—virtually all applications today are built using some form of user interface tool. In this article, we consider cases of both success and failure in past user interface tools. From these cases we extract a set of themes which can serve as lessons for future work. Using these themes, past tools can be characterized by what aspects of the user interface they addressed, their threshold and ceiling, what path of least resistance they offer, how predictable they are to use, and whether they addressed a target that became irrelevant. We believe the lessons of these past themes are particularly important now, because increasingly rapid technological changes are likely to significantly change user interfaces. We are at the dawn of an era where user interfaces are about to break out of the “desktop” box where they have been stuck for the past 15 years. The next millenium will open with an increasing diversity of user interface on an increasing diversity of computerized devices. These devices include hand-held personal digital assistants (PDAs), cell phones, pages, computerized pens, computerized notepads, and various kinds of desk and wall size-computers, as well as devices in everyday objects (such as mounted on refridgerators, or even embedded in truck tires). The increased connectivity of computers, initially evidenced by the World Wide Web, but spreading also with technologies such as personal-area networks, will also have a profound effect on the user interface to computers. Another important force will be recognition-based user interfaces, especially speech, and camera-based vision systems. Other changes we see are an increasing need for 3D and end-user customization, programming, and scripting. All of these changes will require significant support from the underlying user interface sofware tools.

Programming-by-demonstration (PBD) can be used to create tools and methods that eliminate the need to learn difficult computer languages. Gamut is a PBD tool that nonprogrammers can use to create a broader range of interactive software, including games, simulations, and educational software, than they can with other PBD tools. To do this, Gamut provides advanced interaction techniques that make it easier for a developer to express all aspects of an application. These techniques include a simplified way to demonstrate new examples, called “nudges, ” and a way to highlight objects to show they are important. Also, Gamut includes new objects and metaphors like the deck-of-cards metaphor for demonstrating collections of objects and randomness, guide objects for demonstrating relationships that the system would find too difficult to guess, and temporal ghosts which simplify showing relationships with the recent past. These techniques were tested in a formal setting with nonprogrammers to evaluate their effectiveness.

Our research addresses the problem of error correction in speech user interfaces. Previous work hypothesized that switching modality could speed up interactive correction of recognition errors (so-called multimodal error correction). We present a user study that compares, on a dictation task, multimodal error correction with conventional interactive correction, such as speaking again, choosing Tom a list, and keyboard input. Results show that multimodal correction is faster than conventional correction without keyboard input, but slower than correction by typing for users with good typing skills. Furthermore, while users initially prefer speech, they learn to avoid ineffective correction modalities with experience. To extrapolate results from this user study we developed a performance model of multimodal interaction that predicts input speed including time needed for error correction. We apply the model to estimate the impact of recognition technology improvements on correction speeds and the influence of recognition accuracy and correction method on the productivity of dictation systems. Our model is a first step towards formalizing multimodal (recognition-based) interaction.

This paper describes a new technique for transferring data between computers, the synchronized clipboard. Multiple computers can share a synchronized clipboard for all clipboard operations, so that data copied to the clipboard from one computer, using the standard Copy command, can be pasted directly on another computer using the standard Paste command. Synchronized clipboards are well-suited for a single user moving data among several computers in close proximity. We describe an implementation of synchronized clipboards that works across a wide range of existing systems, including 3Com PalmPilots, Microsoft Windows PCs, Unix workstations, and other Java-capable platforms. Our implementation adds no noticeable overhead to local copy and paste operations.

Writing scripts (often called "macros") can be helpful for automating repetitive tasks.Scripting facilities for text editors like Emacs and Microsoft Word have been widely used and available.However, for graphical applications, scripting has been tried many times but has never been successful.This is mainly due to the data description problem of determining how to generalize the particular objects selected at demonstration time.Previous systems have mostly tried to solve this using inferencing, but this has a number of problems, including guessing wrong and providing appropriate feedback and control to users.Therefore, the Topaz framework does not use inferencing and instead allows the user to specify how the appropriate objects should be found, This is achieved by recording changes to which objects are selected and searches for objects, so that scripts can be written with respect to the selected object, in the same way as Emacs keyboard macros.Furthermore, all values can be explicitly generalized in a number of ways, and scripts can be invoked as a result of other commands.By leveraging off of Amulet's command object architecture, programmers get these capabilities for free in their applications.The result is that much more sophisticated scripting capabilities available in applications with no extra work for programmers.

The Pebbles research project is investigating how the functionality and the user interface can be spread across all of the computing and input / output devices that one or more users have available. In this way, the hand-held devices can be used to augment the other computers rather than just being a replacement when other computers are not available. As part of this research, we are looking at how multiple people can use their hand-helds to share control of a main computer running either legacy applications, or custom multi-person applications. We are also looking at how information and control can be fluidly moved between the main screen and the individual hand-held screens.

By combining the strengths of multiple interaction techniques and inferencing algorithms, Gamut can infer behaviors from examples that previously required a developer to annotate or otherwise modify code by hand.Gamut is a programming-by-demonstration (PBD) tool for building whole applications.It revises code automatically when new examples are demonstrated using a recursive procedure that efficiently scans for the differences between a new example and the original behavior.Differences that cannot be resolved by generating a suitable description are handled by another AI algorithm, decision tree learning, providing a significantly greater ability to infer complex relationships.Gamut's interaction techniques facilitate demonstrating many examples quickly and allow the user to give the system hints that show relationships that would be too time consuming to discover by search alone.Altogether, the concepts combined in Gamut will allow nonprogrammers to build software they never could before.

No abstract available.

An open question in the area of Programming by Demonstration (PBD) is how to best represent the inferred program. Without a way to view, edit, and share programs, PBD systems will never reach their full potential. We designed and implemented two graphical representation languages for a PBD desktop similar to the Apple Macintosh Finder. Although a user study showed that both languages enabled nonprogrammers to generate and comprehend programs, the study also revealed that the language that more closely reflected the desktop domain doubled users' abilities to accurately generate programs. Trends suggest that the same language was easier for users to comprehend. These findings suggest that it is possible for a PBD system to enable nonprogrammers to construct programs and that the form of the representation can impact the PBD system's effectiveness. A paper-and-pencil evaluation of the two versions of the PBD desktop prior to the study supported these finding and provided interesting feedback on the interaction between usability evaluations and user studies. In particular, the comparison of the paper-and-pencil evaluation with the empirical evaluation suggested that nonempirical evaluation techniques can provide guidance into how to interpret empirical data and, in particular, that PBD systems need to provide support for programming-strategy selection in order to be successful.

The Amulet user interface development environment makes it easier for programmers to create highly interactive, graphical user interface software for Unix, Windows and the Macintosh. Amulet uses new models for objects, constraints, animation, input, output, commands, and undo. The object system is a prototype instance model in which there is no distinction between classes and instances or between methods and data. The constraint system allows any value of any object to be computed by arbitrary code and supports multiple constraint solvers. Animations can be attached to existing objects with a single line of code. Input from the user is handled by "interactor" objects which support reuse of behavior objects. The output model provides a declarative definition of the graphics and supports automatic refresh. Command objects encapsulate all of the information needed about operations, including support for various ways to undo them. A key feature of the Amulet design is that all graphical objects and behaviors of those objects are explicitly represented at run time, so the system can provide a number of high level built-in functions, including automatic display and editing of objects, and external analysis and control of interfaces. Amulet integrates these capabilities in a flexible and effective manner.

The Amulet user interface development environment uses hierarchical command objects to support the creation of highly-interactive graphical user interfaces.When input arrives or a widget is operated by the user, instead of invoking a call-back procedure as in most other toolkits, Amulet allocates a command object and calls its DO method.Unlike previous uses of command objects, Amulet organizes the commands into a hierarchy, so that low-level operations like dragging or selection invoke low-level commands, which in turn might invoke widget-level commands, which invoke high-level, application-specific commands, and so on.The top-level commands correspond to semantic actions of the program.The result is better modularization because different levels of the user interface are independent, and better code reuse because the lower-level commands, and even many high-level commands such as cut, copy, paste, text edit, and change-color, can be reused from the library.Furthermore, the commands in Amulet support a new form of Undo, where the user can select any previous operation and selectively undo it, repeat it on the same objects, or repeat it on new objects.In addition, operations like scrolling and selections can be undone or repeated, which can be very useful.Thus, the command objects in Amulet make it easier for developers by providing more reusable components, while at the same time providing new capabilities for users.

repair are very accurate.N-best choice appears to be more effective in this setting, probably due to the small vocabularies.

Adding animation to interfaces is a very difficult task with today's toolkits, even though there are many situations in which it would be useful and effective. The Amulet toolkit contains a new form of animation constraint that allows animations to be added to interfaces extremely easily without changing the logic of the application or the graphical objects themselves. An animation constraint detects changes to the value of the slot to which it is attached, and causes the slot to instead take on a series of values interpolated between the original and new values. The advantage over previous approaches is that animation constraints provide significantly better modularity and reuse. The programmer has independent control over the graphics to be animated, the start and end values of the animation, the path through value space, and the timing of the animation. Animations can be attached to any object, even existing widgets from the toolkit, and any type of value can be animated: scalars, ...

Current interactive user interface construction tools are often more of a hindrance than a benefit during the early stages of user interface design. These tools take too much time to use and force designers to specify more of the design details than they wish at this early stage. Most interface designers, especially those who have a background in graphic design, prefer to sketch early interface ideas on paper or on a whiteboard. We are developing an interactive tool called SILK that allows designers to quickly sketch an interface using an electronic pad and stylus. SILK preserves the important properties of pencil and paper: a rough drawing can be produced very quickly and the medium is very flexible. However, unlike a paper sketch, this electronic sketch is interactive and can easily be modified. In addition, our system allows designers to examine, annotate, and edit a complete history of the design. When the designer is satisfied with this early prototype, SILK can transform the sket...

Almost as long as there have been user interfaces, there have been special software systems and tools to help design and implement the user interface software. Many of these tools have demonstrated significant productivity gains for programmers, and have become important commercial products. Others have proven less successful at supporting the kinds of user interfaces people want to build. This article discusses the different kinds of user interface software tools, and investigates why some approaches have worked and others have not. Many examples of commercial and research systems are included. Finally, current research directions and open issues in the field are discussed.

The Lapidary interface design tool is a demonstrational system that allows the graphics and run-time behaviors that go inside an application window to be specified pictorially. In particular, Lapidary allows the designer to draw example pictures of application-specific graphical objects that the end user will manipulate (such as boxes, arrows, or elements of a list), the feedback that shows which objects are selected (such as small boxes on the sides and corners of an object), and the dynamic feedback objects (such as hairline boxes to show where an object is being dragged). The run-time behavior of all these objects can be specified ina straightforward way using constraints, demonstration, and dialog boxes that allow the designer to provide abstract descriptions of the interactive response to the input devices. Lapidary generalizes from these specific example pictures and behaviors to create prototype objects and behaviors from which instances can be made at run-time. A novel feature of Lapidary's implementation is its use of constraints that have been explicitly specified by the designer to help it generalize example objects and behaviors and to guide it in making inferences.

Article Free Access Share on Model-based user interfaces: what are they and why should we care? Authors: Noi Sukaviriya GVU Center, Georgia Institute of Technology, Atlanta, GA GVU Center, Georgia Institute of Technology, Atlanta, GAView Profile , Srdjan Kovacevic US WEST Technologies, 4001 Discovery Dr., Boulder, CO US WEST Technologies, 4001 Discovery Dr., Boulder, COView Profile , James D. Foley GVU Center, Georgia Institute of Technology, Atlanta, GA GVU Center, Georgia Institute of Technology, Atlanta, GAView Profile , Brad A. Myers Computer Science Department, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA Computer Science Department, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PAView Profile , Dan R. Olsen Computer Science Department, Brigham Young University, Provo, UT Computer Science Department, Brigham Young University, Provo, UTView Profile , Matthias Schneider-Hufschmidt Siemens AG, ZFE ST SN 51, D-81730, Munich Siemens AG, ZFE ST SN 51, D-81730, MunichView Profile Authors Info & Claims UIST '94: Proceedings of the 7th annual ACM symposium on User interface software and technologyNovember 1994Pages 133–135https://doi.org/10.1145/192426.192479Published:02 November 1994Publication History 13citation532DownloadsMetricsTotal Citations13Total Downloads532Last 12 Months42Last 6 weeks4 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Pointer variables have long been considered useful for constructing and manipulating data structures in traditional programming languages. This article discusses how pointer variables can be integrated into one-way constraint models and indicates how these constraints can be usefully employed in user interfaces. Pointer variables allow constraints to model a wide array of dynamic application behavior, simplify the implementation of structured objects and demonstrational systems, and improve the storage and efficiency of constraint-based applications. This article presents two incremental algorithms—one lazy and one eager— for solving constraints with pointer variables. Both algorithms are capable of handling (1) arbitrary systems of one-way constraints, including constraints that involve cycles, and (2) editing models that allow multiple changes between calls to the constraint solver. These algorithms are fault tolerant in that they can handle and recover gracefully from formulas that crash due to programmer error. Constraints that use pointer variables have been implemented in a comprehensive user interface toolkit, Garnet, and our experience with applications written in Garnet have proven the usefulness of pointer variable constraints. Many large-scale applications have been implemented using these constraints.

Article Free Access Share on From research prototypes to usable, useful systems: lessons learned in the trenches Authors: Gurminder Singh Institute of Systems Science, National University of Singapore, Kent Ridge, Singapore 0511 Institute of Systems Science, National University of Singapore, Kent Ridge, Singapore 0511View Profile , Mark Linton Silicon Graphics, Inc. Silicon Graphics, Inc.View Profile , Brad A. Myers Computer Science Department, Carnegie Mellon University Computer Science Department, Carnegie Mellon UniversityView Profile , Marti Szczur NASA/Goddard Space Flight Center NASA/Goddard Space Flight CenterView Profile Authors Info & Claims UIST '93: Proceedings of the 6th annual ACM symposium on User interface software and technologyDecember 1993 Pages 139–143https://doi.org/10.1145/168642.168656Online:01 December 1993Publication History 1citation323DownloadsMetricsTotal Citations1Total Downloads323Last 12 Months4Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Plus que les performances d'un systeme technologique, ce sont les possibilites de communication et leurs resultats qui peuvent etre ameliores. L'auteur developpe ici des mecanismes d'entree ou de sortie des donnees qui permettraient une meilleure communication ordinateur-utilisateur.

Abstract

This paper reports on the results of a survey of user interface programming. The survey was widely distributed, and we received 74 responses. The results show that in today's applications, an average of 48% of the code is devoted to the user interface portion. The average time spent on the user interface portion is 45% during the design phase, 50% during the implementation phase, and 37% during the maintenance phase. 34% of the systems were implemented using a toolkit, 27% used a UIMS, 14% used an interface builder, and 26% used no tools. This appears to be because the toolkit systems had more sophisticated user interfaces. The projects using UIMSs or interface builders spent the least percent of time and code on the user interface (around 41%) suggesting that these tools are effective. In general, people were happy with the tools they used, especially the graphical interface builders. The most common problems people reported when developing a user interface included getting users' requirements, writing help text, achieving consistency, learning how to use the tools, getting acceptable performance, and communicating among various parts of the program.

article Free Access Share on Declarative programming in a prototype-instance system: object-oriented programming without writing methods Authors: Brad A. Myers View Profile , Dario A. Giuse View Profile , Brad Vander Zanden View Profile Authors Info & Claims ACM SIGPLAN NoticesVolume 27Issue 10Oct. 1992 pp 184–200https://doi.org/10.1145/141937.141953Online:31 October 1992Publication History 37citation558DownloadsMetricsTotal Citations37Total Downloads558Last 12 Months14Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Conventional interface builders allow the user interface designer to select widgets such as menus, buttons and scroll bars, and lay them out using a mouse. Although these are conceptually simple to use, in practice there are a number of problems. First, a typical widget will have dozens of properties which the designer might change. Insuring that these properties are consistent across multiple widgets in a dialog box and multiple dialog boxes in an application can be very difficult. Second, if the designer wants to change the properties, each widget must be edited individually. Third, getting the widgets laid out appropriately in a dialog box can be tedious. Grids and alignment commands are not sufficient. This paper describes Graphical Tabs and Graphical Styles in the Gild interface builder which solve all of these problems. A “graphical tab” is an absolute position in a window. A “graphical style” incorporates both property and layout information, and can be defined by example, named, applied to other widgets, edited, saved to a file, and read from a file. If a graphical style is edited, then all widgets defined using that style are modified. In addition, because appropriate styles are inferred, they do not have to be explicitly applied.

Many modem user interface development environments use constraints to connect graphical objects.Constraints are relationships that are de&red once and then maintained by the system.Often, systems provide graphical, iconic, or demonstrational techniques for specifying some constraints, but these are incapable of expressing all desired relationships, and it is always necessary to allow the user interface designer to write code to specify complex constraints.The spreadsheet interface described here, called C32, provides the programmer with the full power of writing constraint code in the underlying programming language, but it is significantly easier to use.Unlike other spreadsheets tools for gmphics, C32 automatically generates appropriate object references from mouse clicks in graphics windows and uses inferencing and demonstrational techniques to make constructing and copying constraints easier.In addition, C32 also supports monitoring and debugging interfaces by watching values in the spreadsheet while the user interface is running.

In text formatters such as troff, Scribe, and TEX, users write macro procedures to specify the desired visual appearance.In What-You-See-Is-What-You-Get text formatters, such as MacWrite and Microsoft Word, the formatting is specified by directly manipulating the text.However, some important functionality is lost in these systems since they are not programmable, For example, if the user wants to change the formatting and content of all the chapter headings or page headings, each one must be individually edited.If they had been generated by macros, then editing the macro definition would change them all at once.This paper describes the design for a demonstrational text formatter that allows the user to directly manipulate the formatting of one example, and then the system automatically creates the macro by generalizing the example.This technique makes the formatting for headers, itemized lists, tables, bibliographic references, and many other parts of documents significantly easier to specify and edit.

Article Demonstrational interfaces: Coming soon? Share on Authors: Brad A. Myers School of Computer Science, Carnegie Mellon University, Pittsburgh, PA School of Computer Science, Carnegie Mellon University, Pittsburgh, PAView Profile , Allen Cypher Apple Computer, Inc. Apple Computer, Inc.View Profile , David Maulsby University of Calgary University of CalgaryView Profile , David C. Smith Apple Computer, Inc. Apple Computer, Inc.View Profile , Ben Shneiderman University of Maryland University of MarylandView Profile Authors Info & Claims CHI '91: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsApril 1991 Pages 393–396https://doi.org/10.1145/108844.108966Published:01 March 1991 10citation335DownloadsMetricsTotal Citations10Total Downloads335Last 12 Months4Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

No abstract available.

Article Free Access Share on A nose gesture interface device: extending virtual realities Authors: Tyson R. Henry Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZ Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZView Profile , Scott E. Hudson Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZ Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZView Profile , Andrey K. Yeatts Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZ Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZView Profile , Brad A. Myers School of Computer Science, Carnegie Mellon University, Pittsburgh, PA School of Computer Science, Carnegie Mellon University, Pittsburgh, PAView Profile , Steven Feiner Department of Computer Science, Columbia University, New York, NY Department of Computer Science, Columbia University, New York, NYView Profile Authors Info & Claims UIST '91: Proceedings of the 4th annual ACM symposium on User interface software and technologyNovember 1991 Pages 65–68https://doi.org/10.1145/120782.120789Published:11 November 1991Publication History 5citation995DownloadsMetricsTotal Citations5Total Downloads995Last 12 Months76Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Article Separating application code from toolkits: eliminating the spaghetti of call-backs Share on Author: Brad A. Myers School of Computer Science, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA School of Computer Science, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PAView Profile Authors Info & Claims UIST '91: Proceedings of the 4th annual ACM symposium on User interface software and technologyNovember 1991 Pages 211–220https://doi.org/10.1145/120782.120805Online:11 November 1991Publication History 78citation520DownloadsMetricsTotal Citations78Total Downloads520Last 12 Months18Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Graphical tools are increasingly using constraints to specify the graphical layout and behavior of many parts of an application.However, conventional constraints directly encode the objects they reference, and thus cannot provide support for the dynamic rttntime creation and manipulation of application objects.This paper discusses an extension to current constraint models that allows constraints to indirectly reference objects through pointer variables.Pointer variables permit programmers to create the constraint equivalent of procedures in traditional programming languages.This procedural abstraction allows constraints to model a wide array of dynamic application behavior, simplifies the implementation of structured object and demonstrational systems, and improves the storage and efficiency of highly interactive, graphical applications.It also promotes a simpler, more effective style of programming than conventional constraints.Constraints that use pointer variables are powerful enough to allow a comprehensive user interface toolkit to be built for the first time on top of a constraint system.

Jade is a new interactive tool that automatically creates graphical input dialogs such as dialog boxes and menus. Application programmers write a textual specification of a dialog's contents. This specification contains absolutely no graphical information and thus is look-and-feel independent. The graphic artist uses a direct manipulation graphical editor to define the rules, graphical objects, interaction techniques, and decorations that will govern the dialog's look-and-feel, and stores the results in a look and feel database. Jade combines the application programmer's specification with the look-and-feel database to automatically generate a graphical dialog. If necessary, the graphic artist can then edit the resulting dialog using a graphical editor and these edits will be remembered by Jade, even if the original textual specification is modified. By eliminating all graphical references from the dialog's content specification, Jade requires only the absolutely minimum specification from the application programmer. This also allows a dialog box or menu's look and feel to be rapidly and effortlessly changed by simply switching look and feel databases. Finally, Jade permits complex inter-field relationships to be specified in a simple manner.

Although there has been important progress in models and packages for theoutputof graphics to computer screens, there has been little change in the way thatinputfrom the mouse, keyboard, and other input devices is handled. New graphics standards are still using a fifteen-year-old model even though it is widely accepted as inadequate, and most modern window managers simply return a stream of low-level, device-dependent input events. This paper presents a new model that handles input devices for highly interactive, direct manipulation, graphical user interfaces, which could be used in future toolkits, window managers, and graphics standards. This model encapsulates interactive behaviors into a few “Interactor” object types. Application programs can then create instances of these Interactor objects which hide the details of the underlying window manager events. In addition, Interactors allow a clean separation between the input handling, the graphics, and the application programs. This model has been extensively used as part of the Garnet system and has proven to be convenient, efficient, and easy to learn.

Peridot is an experimental tool that allows designers to create user interface components without conventional programming. The designer draws pictures of what the interface should look like and then uses the mouse and other input devices to demonstrate how the interface should operate. Peridot generalizes from these example pictures and actions to create parameterized procedures, such as those found in conventional user interface libraries such as the Macintosh Toolbox. Peridot uses visual programming, programming by example, constraints, and plausible inferencing to allow nonprogrammers to create menus, buttons, scroll bars, and many other interaction techniques easily and quickly. Peridot created its own interface and can create almost all of the interaction techniques in the Macintosh Toolbox. Therefore, Peridot demonstrates that it is possible to provide sophisticated programming capabilities to nonprogrammers in an easy-to-use manner and still have sufficient power to generate interesting and useful programs.

Although there has been important progress in models and packages for the output of graphics to computer screens, there has been little change in the way that input from the mouse, keyboard and other input devices is handled. New graphics standards are still using a ten year old model even though it is widely accepted as inadequate, and most modern window managers simply return a stream of device-dependent input events. This paper presents a new model for how input devices can be handled for highly-interactive, direct manipulation, graphical user interfaces. This model encapsulates interactive behaviors into a few “interactor” object types. Application programs can then create instances of these interactor objects, and the details of the handling of the input devices are separated from the application and from the output graphics.

The Lapidary user interface tool allows all pictorial aspects of programs to be specified graphically. In addition, the behavior of these objects at run-time can be specified using dialogue boxes and by demonstration. In particular, Lapidary allows the designer to draw pictures of application-specific graphical objects which will be created and maintained at run-time by the application. This includes the graphical entities that the end user will manipulate (such as the components of the picture), the feedback that shows which objects are selected (such as small boxes on the sides and corners of an object), and the dynamic feedback objects (such as hair-line boxes to show where an object is being dragged). In addition, Lapidary supports the construction and use of “widgets” (sometimes called interaction techniques or gadgets) such as menus, scroll bars, buttons and icons. Lapidary therefore supports using a pre-defined library of widgets, and defining a new library with a unique “look and feel.” The run-time behavior of all these objects can be specified in a straightforward way using constraints and abstract descriptions of the interactive response to the input devices. Lapidary generalizes from the specific example pictures to allow the graphics and behaviors to be specified by demonstration.

One of the most difficult aspects of creating graphical, direct manipulation user interfaces is managing the relationships between the graphical objects on the screen and the application data structures that they represent. Coral (Constraint-based Object-oriented Relations And Language) is a new user interface toolkit under development that uses efficiently-implemented constraints to support these relationships. Using Coral, user interface designers can construct interaction techniques such as menus and scroll bars. More importantly, Coral makes it easy to construct direct-manipulation user interfaces specialized to particular applications. Unlike previous constraint-based toolkits, Coral supports defining constraints in the abstract, and then applying them to different object instances. In addition, it provides iteration constructs where lists of items (such as those used in menus) can be constrained as a group. Coral has two interfaces: a declarative interface that provides a convenient way to specify the desired constraints, and a procedural interface that will allow a graphical user interface management system (UIMS) to automatically create Coral calls.

When creating highly-interactive, Direct Manipulation interfaces, one of the most difficult design and implementation tasks is handling the mouse and other input devices. Peridot, a new User Interface Management System, addresses this problem by allowing the user interface designer to demonstrate how the input devices should be handled by giving an example of the interface in action. The designer uses sample values for parameters, and the system automatically infers the general operation and creates the code. After an interaction is specified, it can immediately be executed and edited. This promotes extremely rapid prototyping since it is very easy to design, implement and modify mouse-based interfaces. Peridot also supports additional input devices such as touch tablets, as well as multiple input devices operating in parallel (such as one in each hand) in a natural, easy to specify manner. This is implemented using active values, which are like variables except that the objects that depend on active values are updated immediately whenever they change. Active values are a straightforward and efficient mechanism for implementing dynamic interactions.

Two experiments were run to investigate two-handed input. The experimental tasks were representative of those found in CAD and office information systems.

There has been a great interest recently in systems that use graphics to aid in the programming, debugging, and understanding of computer programs. The terms “Visual Programming” and “Program Visualization” have been applied to these systems. Also, there has been a renewed interest in using examples to help alleviate the complexity of programming. This technique is called “Programming by Example.” This paper attempts to provide more meaning to these terms by giving precise definitions, and then uses these definitions to classify existing systems into a taxonomy. A number of common unsolved problems with most of these systems are also listed.

It is very time-consuming and expensive to create the graphical, highly-interactive styles of user interfaces that are increasingly common. User Interface Management Systems (UIMSs) attempt to make the creation of user interfaces easier, but most existing UIMSs cannot create the low-level interaction techniques (pop-up pull-down and fixed menus, on-screen "light buttons", scroll-bars, elaborate feedback mechanisms and animations, etc.) that are frequently used. This paper describes Peridot, a system that automatically creates the code for these user interfaces while the designer demonstrates to the system how the interface should look and work. Peridot uses rule-based inferencing so no programming by the designer is required, and Direct Manipulation techniques are used to create Direct Manipulation interfaces, which can make full use of a mouse and other input devices. This allows extremely rapid prototyping of user interfaces.

A “percent-done progress indicator” is a graphical technique which allows the user to monitor the progress through the processing of a task. Progress indicators can be displayed on almost all types of output devices, and can be used with many different kinds of programs. Practical experience and formal experiments show that prograss indicators are an important and useful user-interface tool, and that they enhance the attractiveness and effectiveness of programs that incorporate them. This paper discusses why progress indicators are important. It includes the results of a formal experiment with progress indicators. One part of the experiment demonstrates that people prefer to have progress indicators. Another part attempted to replicate earlier findings to show that people prefer constant to variable response time in general, and then to show that this effect is reversed with progress indicators, but the results were not statistically significant. In fact, no significant preference for constant response time was shown, contrary to previously published results.

Many modern computer languages allow the programmer to define and use a variety of data types. Few programming systems, however, allow the programmer similar flexibility when displaying the data structures for debugging, monitoring and documenting programs. Incense is a working prototype system that allows the programmer to interactively investigate data structures in actual programs. The desired displays can be specified by the programmer or a default can be used. The default displays provided by Incense present the standard form for literals of the basic types, the actual names for scalar types, stacked boxes for records and arrays, and curved lines with arrowheads for pointers. In addition to displaying data structures, Incense also allows the user to select, move, erase and redimension the resulting displays. These interactions are provided in a uniform, natural manner using a pointing device (mouse) and keyboard.