A diversity of user goals and strategies make creation-oriented applications such as word processors or photo-editors difficult to comprehensively test. Evaluating such applications requires testing a large pool of participants to capture the diversity of experience, but traditional usability testing can be prohibitively expensive. To address this problem, this article contributes a new usability evaluation method called backtracking analysis, designed to automate the process of detecting and characterizing usability problems in creation-oriented applications. The key insight is that interaction breakdowns in creation-oriented applications often manifest themselves in backtracking operations that can be automatically logged (e.g., undo and erase operations). Backtracking analysis synchronizes these events to contextual data such as screen capture video, helping the evaluator to characterize specific usability problems. The results from three experiments demonstrate that backtracking events can be effective indicators of usability problems in creation-oriented applications, and can yield a cost-effective alternative to traditional laboratory usability testing.

Digital technologies have enabled new temporalities of media consumption in the home. Through a field study of home television viewing practices, we investigated temporal orderings of television watching. In contrast to traditional pictures of television use, our evidence suggests that rhythms across households play an important role in shaping television watching. Further, we found a flexibility and openness within the patterns of television viewing that we refer to as “plasticity.” Our data suggest that plasticity and rhythms co-exist and together compose the qualitative experience of domestic television time; an understanding of both aspects of temporality suggests an approach for the design of future television technologies.

One approach to reducing the costs of usability testing is to facilitate the automatic detection of critical incidents: serious breakdowns in interaction that stand out during software use. This research evaluates the use of undo and erase events as indicators of critical incidents in Google SketchUp (a 3D-modeling application), measuring an indicator's usefulness by the numbers and types of usability problems discovered. We compared problems identified using undo and erase events to problems identified using the user-reported critical incident technique [Hartson and Castillo 1998]. In a within-subjects experiment with 35 participants, undo and erase episodes together revealed over 90% of the problems rated as severe, several of which would not have been discovered by self-report alone. Moreover, problems found by all three methods were rated as significantly more severe than those identified by only a subset of methods. These results suggest that undo and erase events will serve as useful complements to user-reported critical incidents for low cost usability evaluation of creation-oriented applications like SketchUp.

The Cognitive Walkthrough methodology was developed in an effort to bring cognitive theory closer to practice; to enhance the design and evaluation of use interfaces in industrial settings. For the first time, small teams of professional developers have used this method to critique three complex software systems. In this paper we report evidence about how the methodology worked for these evaluations. We focus on five core issues: (1) task selection, coverage, and evaluation, (2) the process of doing a Cognitive Walkthrough, (3) requisite knowlege for the evaluators, (4) group walkthroughs, and (5) the interpretation of results. Our findings show that many variables can affect the success of the technique; we believe that if the Cognitive Walkthrough is ultimately to be successful in industrial settings, the method must be refined and augmented in a variety of ways.

Article Free Access Share on Consultants and apprentices: observations about learning and collaborative problem solving Authors: Lucy M. Berlin View Profile , Robin Jeffries View Profile Authors Info & Claims CSCW '92: Proceedings of the 1992 ACM conference on Computer-supported cooperative workDecember 1992 Pages 130–137https://doi.org/10.1145/143457.143471Online:01 December 1992Publication History 9citation516DownloadsMetricsTotal Citations9Total Downloads516Last 12 Months15Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

evaluation; guidelines; usability testing; cognitive walkthrough A user interface (UI) for a software product was evaluated prior to its release by four groups, each applying a different technique: heuristic evaluation, software guidelines, cognitive walkthroughs, and usability testing. Heuristic evaluation by several ill specialists found the most serious problems with the least amount of effort, although they also reported a large number of low-priority problems. The relative advantages of all the techniques are discussed, and suggestions for improvements in the techniques are offered.

In the domain of interaction languages, forms have been found to be of value in allowing users, especially non-programmers, to specify objects and operations with a minimum of training, time, and errors. Most of that research, however, has been on the use of data base query languages. The present research found that in a procedural task of specifying mail filtering instructions, non-programmers using a form were as fast as programmers using a procedural language, although programmers using the form were faster still.

Four experiments study the errors students make using LISP functions. The first two experiments show that frequency of errors is increased by increasing the complexity of irrelevant aspects of the problem. The experiments also show that the distribution of errors is largely random and that subjects' errors seem to result from slips rather than from misconceptions. Experiment 3 shows that subjects' errors tend to involve loss of parentheses in answers when the resulting errors are well-formed LISP expressions. Experiment 4 asks subjects, who knew no LISP, to judge the reasonableness of the answers to various LISP function calls. Subjects could detect many errors on the basis of general criteria of what a reasonable answer should look like. On the basis of these four experiments, we conclude that errors occur when there is a loss of information in the working memory representation of the problem and when the resulting answer still looks reasonable.