This paper presents findings from two deployments of an autonomous mobile robot in older adult low income Supportive Apartment Living (SAL) facilities. Design guidelines for the robot hardware and software were based on query of clinicians, caregivers and older adults through focus groups, member checks and surveys, to identify what each group believed to be the most important daily activities for older adults to accomplish physically, mentally and socially. After data analysis, hydration and walking encouragement were found to be critical daily activities, becoming the focus of our deployments. The aim of the deployments was to understand the efficacy of human-robot interaction and identify ways to enhance the robot design and programming. Through observation of older adults interacting with the robot and post-interaction surveys filled out by the older adults, conclusions were drawn for further advancement of the robot development to be tested in future deployments. Results overall indicated high perceived usefulness and growing acceptance of the robot by older adults with increased interactions.

This paper introduces CustomPrograms, a rapid programming system for mobile service robots. With CustomPrograms, roboticists can quickly create new behaviors and try unexplored use cases for commercialization. In our system, the robot has a set of primitive capabilities, such as navigating to a location or interacting with users on a touch screen. Users can then compose these primitives with general-purpose programming language constructs like variables, loops, conditionals, and functions. The programming language is wrapped in a graphical interface. This allows inexperienced or novice programmers to benefit from the system as well. We describe the design and implementation of CustomPrograms on a Savioke Relay robot in detail. Based on interviews conducted with Savioke roboticists, designers, and business people, we learned of several potential new use cases for the robot. We characterize our system's ability to fulfill these use cases. Additionally, we conducted a user study of the interface with Savioke employees and outside programmers. We found that experienced programmers could learn to use the interface and create 3 real-world programs during the 90 minute study. Inexperienced programmers were less likely to create complex programs correctly. We provide an analysis of the errors made during the study, and highlight the most common pieces of feedback we received. Two case studies show how the system was used internally at Savioke and at a major trade show.

Software testing is the most time-intensive and resource-intensive aspect of software development. Can support for testing be improved? This case study describes the motivations and design decisions behind the development of the testing tool, CoTester and its deployment to multiple development teams. CoTester outperforms available testing tools by representing tests using an easy-to-understand scripting language and thus making the tests easily editable. The design decisions of the testing tool were derived after conducting a series of interviews with testers and collecting their experiences with manual as well as automated testing. CoTester was developed to support these users, working in an environment of mixed manual and automatic tests, with a progression from manual to automatic testing when circumstances warrant. A series of deployments to four development teams showed that CoTester worked very well for non-professional testers (i.e. those who do testing only part-time), and it was also found to be useful by some professional testers. Copyright © 2012 John Wiley & Sons, Ltd.

We define the problem of automatic instruction interpretation as follows. Given a natural language instruction, can we automatically predict what an instruction follower, such as a robot, should do in the environment to follow that instruction? Previous approaches to automatic instruction interpretation have required either extensive domain-dependent rule writing or extensive manually annotated corpora. This article presents a novel approach that leverages a large amount of unannotated, easy-to-collect data from humans interacting in a game-like environment. Our approach uses an automatic annotation phase based on artificial intelligence planning, for which two different annotation strategies are compared: one based on behavioral information and the other based on visibility information. The resulting annotations are used as training data for different automatic classifiers. This algorithm is based on the intuition that the problem of interpreting a situated instruction can be cast as a classification problem of choosing among the actions that are possible in the situation. Classification is done by combining language, vision, and behavior information. Our empirical analysis shows that machine learning classifiers achieve 77% accuracy on this task on available English corpora and 74% on similar German corpora. Finally, the inclusion of human feedback in the interpretation process is shown to boost performance to 92% for the English corpus and 90% for the German corpus.

Online communities are increasingly being deployed in enterprises to increase productivity and share expertise. Community leaders are critical for fostering successful communities, but existing technologies rarely support leaders directly, both because of a lack of clear data about leader needs, and because existing tools are member- rather than leader-centric. We present the evidence-based design and evaluation of a novel tool for community leaders, Community Insights (CI). CI provides actionable analytics that help community leaders foster healthy communities, providing value to both members and the organization. We describe empirical and system contributions derived from a long-term deployment of CI to leaders of 470 communities over 10 months. Empirical contributions include new data showing: (a) which metrics are most useful for leaders to assess community health, (b) the need for and how to design actionable metrics, (c) the need for and how to design contextualized analytics to support sensemaking about community data. These findings motivate a novel community system that provides leaders with useful, actionable and contextualized analytics.

Task automation systems promise to increase human productivity by assisting us with our mundane and difficult tasks. These systems often rely on people to (1) identify the tasks they want automated and (2) specify the procedural steps necessary to accomplish those tasks (i.e., to create task models). However, our interviews with users of a Web task automation system reveal that people find it difficult to identify tasks to automate and most do not even believe they perform repetitive tasks worthy of automation. Furthermore, even when automatable tasks are identified, the well-recognized difficulties of specifying task steps often prevent people from taking advantage of these automation systems. In this research, we analyze real Web usage data and find that people do in fact repeat behaviors on the Web and that automating these behaviors, regardless of their complexity, would reduce the overall number of actions people need to perform when completing their tasks, potentially saving time. Motivated by these findings, we developed LiveAction, a fully-automated approach to generating task models from Web usage data. LiveAction models can be used to populate the task model repositories required by many automation systems, helping us take advantage of automation in our everyday lives.

As applications are developed, functional tests ensure they continue to function as expected. Nowadays, functional testing is mostly done manually, with human testers verifying a system's functionality themselves, following hand-written instructions. While there exist tools supporting functional test automation, in practice they are hard to use, require programming skills, and do not provide good support for test maintenance. In this paper, we take an alternative approach: we semi-automatically convert hand-written instructions into automated tests. Our approach consists of two stages: first, employing machine learning and natural language processing to compute an intermediate representation from test steps; and second, interactively disambiguating that representation to create a fully automated test. These two stages comprise a complete system for converting hand-written functional tests into automated tests. We also present a quantitative study analyzing the effectiveness of our approach. Our results show that 70% of manual test steps can be automatically converted to automated test steps with no user intervention.

New enterprise tools (wikis, team spaces, social tags) offer potential benefits for enterprise collaboration, providing shared resources to organize work. However, a vast amount of collaboration still takes place by email. But email is problematic for collaboration because information may be distributed across multiple messages in an overloaded inbox. Email also increases workload as each individual has to manage their own versions of collaborative materials. We present a novel system, Topika that integrates email with collaboration tools. It allows users to continue to use email while also enjoying the benefits of these dedicated tools. When a user composes an email Topika analyzes the message and suggests relevant shared spaces (e.g., wiki pages) within the user's collaboration tools. This allows her to post the email to those spaces. An evaluation of Topika's suggestion algorithm shows that it performs well at accurately suggesting shared spaces.

With all the information available on the web, there is a growing need to provide mobile access to this information for the large, growing population of mobile internet users. In this paper, we propose a solution to the problem of open web mobile information retrieval, by conducting a dialogue with the user over a simple text-based interface. Using techniques from NLP, web page analysis, and information extraction, our approach automatically navigates web sites on the user's behalf and extracts specific information from those sites to present to the user textually. Empirical evaluation shows that our approach to open web information retrieval is feasible, and a qualitative evaluation validates that such a system meets user needs for mobile information access.

ActionShot is an integrated web browser tool that creates a fine-grained history of users' browsing activities by continually recording their browsing actions at the level of interactions, such as button clicks and entries into form fields. ActionShot provides interfaces to facilitate browsing and searching through this history, sharing portions of the history through established social networking tools such as Facebook, and creating scripts that can be used to repeat previous interactions at a later time. ActionShot can also create short textual summaries for sequences of interactions. In this paper, we describe the ActionShot and our initial explorations of the tool through field deployments within our organization and a lab study. Overall, we found that ActionShot's history features provide value beyond typical browser history interfaces.

In this paper, we present CoTester, a system designed to decrease the difficulty of testing web applications. CoTester allows testers to create test scripts that are represented in an easy-to-understand scripting language rather than a complex programming language, which allows tests to be created rapidly and by non-developers. CoTester improves the management of test scripts by grouping sequences of lowlevel actions into subroutines, such as "log in" or "check out shopping cart", which help testers visualize test structure and make bulk modifications. A key innovation in CoTester is its ability to automatically identify these subroutines using a machine learning algorithm. Our algorithm is able to achieve 91% accuracy at recognizing a set of 7 representative subroutines commonly found in test scripts.

This paper presents CoCo, a system that automates web tasks on a user's behalf through an interactive conversational interface. Given a short command such as "get road conditions for highway 88," CoCo synthesizes a plan to accomplish the task, executes it on the web, extracts an informative response, and returns the result to the user as a snippet of text. A novel aspect of our approach is that we leverage a repository of previously recorded web scripts and the user's personal web browsing history to determine how to complete each requested task. This paper describes the design and implementation of our system, along with the results of a brief user study that evaluates how likely users are to understand what CoCo does for them.

For blind web users, completing tasks on the web can be frustrating. Each step can require a time-consuming linear search of the current web page to find the needed interactive element or piece of information. Existing interactive help systems and the playback components of some programming-by-demonstration tools identify the needed elements of a page as they guide the user through predefined tasks, obviating the need for a linear search on each step. We introduce TrailBlazer, a system that provides an accessible, non-visual interface to guide blind users through existing how-to knowledge. A formative study indicated that participants saw the value of TrailBlazer but wanted to use it for tasks and web sites for which no existing script was available. To address this, TrailBlazer offers suggestion-based help created on-the-fly from a short, user-provided task description and an existing repository of how-to knowledge. In an evaluation on 15 tasks, the correct prediction was contained within the top 5 suggestions 75.9% of the time.

Mashups are an increasingly popular way to integrate data from multiple web sites to fit a particular need, but it often requires substantial technical expertise to create them. To lower the barrier for creating mashups, we have extended the CoScripter web automation tool with a spreadsheet-like environment called Vegemite. Our system uses direct-manipulation and programming-by-demonstration tech-niques to automatically populate tables with information collected from various web sites. A particular strength of our approach is its ability to augment a data set with new values computed by a web site, such as determining the driving distance from a particular location to each of the addresses in a data set. An informal user study suggests that Vegemite may enable a wider class of users to address their information needs.

Modern enterprises are replete with numerous online processes. Many must be performed frequently and are tedious, while others are done less frequently yet are complex or hard to remember. We present interviews with knowledge workers that reveal a need for mechanisms to automate the execution of and to share knowledge about these processes. In response, we have developed the CoScripter system (formerly Koala [11]), a collaborative scripting environment for recording, automating, and sharing web-based processes. We have deployed CoScripter within a large corporation for more than 10 months. Through usage log analysis and interviews with users, we show that CoScripter has addressed many user automation and sharing needs, to the extent that more than 50 employees have voluntarily incorporated it into their work practice. We also present ways people have used CoScripter and general issues for tools that support automation and sharing of how-to knowledge.

Today's web pages provide many useful features, but unfortunately nearly all are designed first and foremost for the desktop form factor. At the same time, the number of mobile devices with different form factors and unique input and output facilities is growing substantially. The Highlight environment addresses these problems by allowing users to start with existing sites they already use and create mobile versions that are customized to their tasks and mobile devices. This re-authoring is performed through a combination of demonstrating desired interactions with an existing web site and directly specifying content to be included on mobile pages. The system has been tested successfully with a variety of existing sites. A study showed that novice users were able to use the system to create useful mobile applications for sites of their own choosing.

We present Koala, a system that enables users to capture, share, automate, and personalize business processes on the web. Koala is a collaborative programming-by-demonstration system that records, edits, and plays back user interactions as pseudo-natural language scripts that are both human- and machine-interpretable. Unlike previous programming by demonstration systems, Koala leverages sloppy programming that interprets pseudo-natural language instructions (as opposed to formal syntactic statements) in the context of a given web page's elements and actions. Koala scripts are automatically stored in the Koalescence wiki, where a community of users can share, run, and collaboratively develop their "how-to" knowledge. Koala also takes advantage of corporate and personal data stores to automatically generalize and instantiate user-specific data, so that scripts created by one user are automatically personalized for others. Our initial experiences suggest that Koala is surprisingly effective at interpreting instructions originally written for people.

We studied files stored by members of a work organization for patterns of social commonality. Discovering identical or similar documents, applications, developer libraries, or other files may suggest shared interests or experience among users. Examining actual file data revealed a number of individual and aggregate practices around file storage. For example, pairs of users typically have many (over 13,000) files in common. A prototype called LiveWire exploits this commonality to make file backup and restore more efficient for a work organization. We removed commonly shared files and focused on specific filetypes that represent user activity to find more meaningful files in common. The Consolidarity project explores how patterns of file commonality could encourage social networking in an organizational context. Mechanisms for addressing the privacy concerns raised by this approach are discussed.

Employee directories play a valuable role in helping people find others to collaborate with, solve a problem, or provide needed expertise. Serving this role successfully requires accurate and up-to-date user profiles, yet few users take the time to maintain them. In this paper, we present a system that enables users to tag other users with key words that are displayed on their profiles. We discuss how people-tagging is a form of social bookmarking that enables people to organize their contacts into groups, annotate them with terms supporting future recall, and search for people by topic area. In addition, we show that people-tagging has a valuable side benefit: it enables the community to collectively maintain each others' interest and expertise profiles. Our user studies suggest that people tag other people as a form of contact management and that the tags they have been given are accurate descriptions of their interests and expertise. Moreover, none of the people interviewed reported offensive or inappropriate tags. Based on our results, we believe that peopletagging will become an important tool for relationship management in an organization.

Email-based activity management systems promise to give users better tools for managing increasing volumes of email, by organizing email according to a user's activities. Current activity management systems do not automatically classify incoming messages by the activity to which they belong, instead relying on simple heuristics (such as message threads), or asking the user to manually classify incoming messages as belonging to an activity. This paper presents several algorithms for automatically recognizing emails as part of an ongoing activity. Our baseline methods are the use of message reply-to threads to determine activity membership and a naïve Bayes classifier. Our SimSubset and SimOverlap algorithms compare the people involved in an activity against the recipients of each incoming message. Our SimContent algorithm uses IRR (a variant of latent semantic indexing) to classify emails into activities using similarity based on message contents. An empirical evaluation shows that each of these methods provide a significant improvement to the baseline methods. In addition, we show that a combined approach that votes the predictions of the individual methods performs better than each individual method alone.

Many structured activities are managed by email. For instance, a consumer purchasing an item from an e-commerce vendor may receive a message confirming the order, a warning of a delay, and then a shipment notification. Existing email clients do not understand this structure, forcing users to manage their activities by sifting through lists of messages. As a first step to developing email applications that provide high-level support for structured activities, we consider the problem of automatically learning an activity's structure. We formalize activities as finite-state automata, where states correspond to the status of the process, and transitions represent messages sent between participants. We propose several unsupervised machine learning algorithms in this context, and evaluate them on a collection of e-commerce email.

Traditional documentation for computer-based procedures is difficult to use: readers have trouble navigating long complex instructions, have trouble mapping from the text to display widgets, and waste time performing repetitive procedures. We propose a new class of improved documentation that we call follow-me documentation wizards. Follow-me documentation wizards step a user through a script representation of a procedure by highlighting portions of the text, as well application UI elements. This paper presents algorithms for automatically capturing follow-me documentation wizards by demonstration, through observing experts performing the procedure. We also present our DocWizards implementation on the Eclipse platform. We evaluate our system with an initial user study that showing that most users have a marked preference for this form of guidance over traditional documentation.

No abstract available.

Technical support procedures are typically very complex. Users often have trouble following printed instructions describing how to perform these procedures, and these instructions are difficult for support personnel to author clearly. Our goal is to learn these procedures by demonstration, watching multiple experts performing the same procedure across different operating conditions, and produce an executable procedure that runs interactively on the user's desktop. Most previous programming by demonstration systems have focused on simple programs with regular structure, such as loops with fixed-length bodies. In contrast, our system induces complex procedure structure by aligning multiple execution traces covering different paths through the procedure. This paper presents a solution to this alignment problem using Input/Output Hidden Markov Models. We describe the results of a user study that examines how users follow printed directions. We present Sheepdog, an implemented system for capturing, learning, and playing back technical support procedures on the Windows desktop. Finally, we empirically evalute our system using traces gathered from the user study and show that we are able to achieve 73% accuracy on a network configuration task using a procedure trained by non-experts.

No abstract available.

An emerging approach to multi-device application development requires developers to build an abstract semantic model that is translated into specific implementations for web browsers, PDAs, voice systems and other user interfaces. Specifying abstract semantics can be difficult for designers accustomed to working with concrete screen-oriented layout. We present an approach to model recovery: inferring semantic models from existing applications, enabling developers to use familiar tools but still reap the benefits of multi-device deployment. We describe MORE, a system that converts the visual layout of HTML forms into a semantic model with explicit captions and logical grouping. We evaluate MOREs performance on forms from existing Web applications, and demonstrate that in most cases the difference between the recovered model and a hand-authored model is under 5%

While existing learning techniques can be viewed as inducing programs from examples, most research has focused on rather narrow classes of programs, e.g., decision trees or logic rules. In contrast, most of today's programs are written in languages such as C++ or Java. Thus, many tasks we wish to automate (e.g. programming by demonstration and software reverse engineering) might be best formulated as induction of code in a procedural language. In this paper we apply version space algebra [10] to learn such procedural programs given execution traces. We consider two variants of the problem (whether or not program-step information is included in the traces) and evaluate our implementation on a corpus of programs drawn from introductory computer science textbooks. We show that our system can learn correct programs from few traces.

Applications of machine learning can be viewed as teacherstudent interactions in which the teacher provides training examples and the student learns a generalization of the training examples. One such application of great interest to the IUI community is adaptive user interfaces. In the traditional learning interface, the scope of teacher-student interactions consists solely of the teacher/user providing some number of training examples to the student/learner and testing the learned model on new examples. Active learning approaches go one step beyond the traditional interaction model and allow the student to propose new training examples that are then solved by the teacher. In this paper, we propose that interfaces for machine learning should even more closely resemble human teacher-student relationships. A teacher's time and attention are precious resources. An intelligent studentmust proactively contribute to the learning process, by reasoning about the quality of its knowledge, collaborating with the teacher, and suggesting new examples for her to solve. The paper describes a varietyof richinteraction modes that enhance the learning process and presents a decision-theoretic framework, called DIAManD, for choosing the best interaction. We apply the framework to the SMARTedit programming by demonstration system and describe experimental validation and preliminary user feedback.

Although Programmingby Demonstration (PBD) has the potential to improve the productivity of unsophisticated users, previous PBD systems have used brittle, heuristic, domain-specific approaches to execution-trace generalization.In this paper we define two applicationindependent methods for performing generalization that are based on well-understood machine learning technology.TGENV~ uses version-space generalization, and TGENFOIL is based on the FOIL inductive logic programming algorithm.We analyze each method both theoretically and empirically, arguing that TGENVS has lower sample complexity, but TGENFOIL can learn a much more interesting class of programs.