Analytical usability evaluation methods (UEMs) can complement empirical evaluation of systems: for example, they can often be used earlier in design and can provide accounts of why users might experience difficulties, as well as what those difficulties are. However, their properties and value are only partially understood. One way to improve our understanding is by detailed comparisons using a single interface or system as a target for evaluation, but we need to look deeper than simple problem counts: we need to consider what kinds of accounts each UEM offers, and why. Here, we report on a detailed comparison of eight analytical UEMs. These eight methods were applied to a robotic arm interface, and the findings were systematically compared against video data of the arm in use. The usability issues that were identified could be grouped into five categories: system design, user misconceptions, conceptual fit between user and system, physical issues, and contextual ones. Other possible categories such as user experience did not emerge in this particular study. With the exception of Heuristic Evaluation, which supported a range of insights, each analytical method was found to focus attention on just one or two categories of issues. Two of the three “home-grown” methods (Evaluating Multimodal Usability and Concept-based Analysis of Surface and Structural Misfits) were found to occupy particular niches in the space, whereas the third (Programmable User Modeling) did not. This approach has identified commonalities and contrasts between methods and provided accounts of why a particular method yielded the insights it did. Rather than considering measures such as problem count or thoroughness, this approach has yielded insights into the scope of each method.

We focus on the ability of two analytical usability evaluation methods (UEMs), namely CASSM (Concept-based Analysis for Surface and Structural Misfits) and Cognitive Walkthrough, to identify usability issues underlying the use made of two London Underground ticket vending machines. By setting both sets of issues against the observed interactions with the machines, we assess the similarities and differences between the issues depicted by the two methods. In so doing we de-emphasise the mainly quantitative approach which is typical of the comparative UEM literature. However, by accounting for the likely consequences of the issues in behavioural terms, we reduced the proportion of issues which were anticipated but not observed (the false positives), compared with that achieved by other UEM studies. We assess these results in terms of the limitations of problem count as a measure of UEM effectiveness. We also discuss the likely trade-offs between field studies and laboratory testing.

Is it possible to achieve some of the benefits of formal testing within the informal programming conventions of the spreadsheet paradigm? We have been working on an approach that attempts to do so via the development of a testing methodology for this paradigm. Our “What You See Is What You Test” (WYSIWYT) methodology supplements the convention by which spreadsheets provide automatic immediate visual feedback about values by providing automatic immediate visual feedback about “testedness”. In previous work we described this methodology; in this paper, we present empirical data about the methodology's effectiveness. Our results show that the use of the methodology was associated with significant improvement in testing effectiveness and efficiency even with no training on the theory of testing or test adequacy that the model implements. These results may be due at least in part to the fact that use of the methodology was associated with a significant reduction in overconfidence.

Although cognitive psychology showed much initial promise, it has failed to make significant contributions to the study of human-computer interaction, which has led to a rejection of cognitivism in favour of situated action theory. The authors accept that the critique has much to offer, but reject the outright abandoning of cognitivism. Cognitive psychology needs a common language in which to describe interaction between people and artifacts: two examples of research in progress are described, one focused on events, the other on representations and the relationship between the information display and the conceptual model. Cognitive psychology also needs a better delivery method than the traditional research paper, and the idea is proposed of a vocabulary of ‘cognitive dimensions’, terms which can be meaningfully used by non-specialists (who will recognise familiar but uncrystallised concepts) and which can be used as indexes to the professional literature. These two components form a proposal for improving the effectiveness of cognitive psychology. The paper ends with the hope that mainstream cognitive psychology will broaden its area of enquiry.

Software engineers have developed sophisticated “object‐oriented” programming environments that are intended to make the reuse of program code easy. Experience has shown that these environments can be improved: Even very experienced programmers have problems in locating and comprehending code for reuse. Programs cannot be modified as readily as had been anticipated. We describe the problems in terms of “cognitive dimensions” of notational systems and show how improved support for opportunistic design may be achieved. A central tenet is that programmers are not at present able to externalize enough of their knowledge about a program. We propose a scheme for attaching a “description level” in which arbitrary attributes and relationships can be recorded in a “browsable” form. Our conclusions stress improving the means for programmers to represent facts rather than the provision of predefined knowledge bases.

A Programmable User Model (PUM) is a psychologically constrained architecture which an interface designer is invited to program to simulate a user performing a range of tasks with a proposed interface. It provides a novel way of conveying psychological considerations to the designer, by involving the designer in the process of making predictions of usability. Development of the idea leads to a complementary perspective, of the PUM as an interpreter for an “instruction language”. The methodology used in this research involves the use of concrete HCI scenarios to assess different approaches to cognitive modelling. The research findings include analyses of the cognitive processes involved in the use of interactive computer systems, and a number of issues to be resolved in future cognitive models.

A formal model of the mental representation of task languages is presented. The model is a metalanguage for defining task-action grammars (TAG): generative grammars that rewrite simple tasks into action specifications. Important features of the model are (a) Identification of the "simple-tasks" that users can perform routinely and that require no control structure; (b) Representation of simple-tasks by collections of semantic components reflecting a categorization of the task world; (c) Marking of tokens in rewrite rules with the semantic features of the task world to supply selection restrictions on the rewriting of simple-tasks into action specifications. This device allows the representation of family resemblances between individual task-action mappings. Simple complexity metrics over task-action grammars make predictions about the relative learnability of different task language designs. Some empirical support for these predictions is derived from the existing empirical literature on command language learning, and from two unreported experiments. Task-action grammars also provide designers with an analytic tool for exposing the configural properties of task languages.

Users perceive consistency and inconsistency in syntax, and family resemblances among syntactic constructions. These factors are not captured in conventional BNF-like grammars. We argue that a generalised form of a two-level grammar is a better model of the user's perceptions, and show how the model relates to current psychological notions of organisation in recall and language learning. The model provides a unified interpretation of many previous results in HCI: we analyse here findings by Reisner (1981) and Barnard et al. (1981). Two preliminary experimental tests supporting the model are described.

Abstract It has become fashionable to criticize nested conditionals, which are supposed to tax the human mind, and to recommend that they should be ‘un‐nested’, turning if‐then‐if into if‐then‐else‐if. Experiment casts doubt on this, however, and suggests that indeed the unnested form is only slightly better in the best conditions and is much worse in unsuitable conditions. Rather than hypothesizing about how the programmer's mind works a much more important principle seems to be to provide perceptual cues that will make program structure manifest rather than latent.

An outline history of the programming language conditional is given, followed by a description of some experiments to find which form of conditional is easier to use. Experiments on whether improved teaching methods and structure aids can help users of programming languages with conditional GOTOs to write and debug their programs more easily are then described. It is concluded that behavioural experiments on programming languages can be useful to the language designer, and that the teaching of desirable program structure, rather than just programming language structure, is useful to the student.