Usability evaluations typically occur throughout the life cycle of a product. A number of decisions and practical biases concerning the tasks selected for usability evaluations can influence the results. A pervasive bias is to select only tasks that are possible to perform with the product under evaluation, introducing a subtle bias for the participants. One way to avoid this problem is to employ user-defined tasks (UDTs) in usability evaluations. In addition, having participants define tasks to perform in a product evaluation allows a more accurate assessment of product usability. This is because UDTs based on users' requirements and expectations should be relatively independent of the functional capabilities of a product. However, there are a number of methodological and practical issues that result from the introduction of UDTs in a usability evaluation. The best approach is to design hybrid evaluations using both UDTs and product-supported tasks.

A study was conducted that used continuous‐vicarious ratings (CVRs) of difficulty to investigate how people make judgments of task difficulty. Twelve people performed five editing tasks using a text editor. After each task, the participants rated the overall difficulty they experienced in performing the task. They then viewed a videotape of their performance. While the participants were viewing the tape, they were asked to rate continuously the difficulty they felt they had experienced at that moment when performing the task. From this curve of difficulty over time, seven variables were obtained as possible candidates in predicting their overall task‐difficulty ratings. It was found that the contrast between the maximum or peak of the continuous rating and the mean difficulty level best accounted for the post‐task ratings of difficulty, followed by the time it took to perform a task. In other words, the more the peak difficulty stood out from the background average‐difficulty level combined with the more time spent performing a task, the higher the participants rated their overall task difficulty. A three‐variable (maximum difficulty, mean difficulty, and time) power‐function model was developed that best predicted the post‐task difficulty ratings. The terms in this model were statistically significant and accounted for 63% of the variability in the task‐difficulty ratings. These results imply that human factors practitioners may be more effective focusing on and improving what people judge to be the most difficult aspects of an interface. Indeed, the results suggest that concentrating on other areas to improve an interface may actually cause an increase in perceived difficulty simply because the problem causing the peak difficulty has become more salient.

There is a desire to streamline the development design process by conducting usability evaluations with fewer subjects. This study examined the impact of using fewer subjects on the accuracy and stability of the mean and median. Monte Carlo simulations were conducted to model the skewed task‐completion times typically found in usability studies. The results showed that the mean was a more accurate estimate of its respective population parameter than the median and should be the preferred metric in usability evaluations. However, the mean cannot always be used because it will underestimate the population mean when there are missing data as a result of subjects exceeding time limits. Alternatively, it was found that the median tended to consistently overestimate the population median by as much as 10% in one examined case. The average 95% confidence limits of both estimates were quite broad, particularly for the smaller sample sizes. Based on these results, it was concluded that with reduced sample sizes (e.g., 5), we are not able to measure population parameters with either a high degree of confidence or, in the case of medians, with any accuracy. It was recommended that usability measurements take a back seat to usability testing when conducting tests with small sample sizes.

Too often Human Factors guidelines or checklists are of limited value. The reasons for following them and the degree to which they should be followed are not always apparent. These problems in conjunction with the fact that they are usually full of words or concepts that are not adequately defined, result in guidelines which are misused and/or grossly under-consulted.