Post-task ratings of difficulty in a usability test have the potential to provide diagnostic information and be an additional measure of user satisfaction. But the ratings need to be reliable as well as easy to use for both respondents and researchers. Three one-question rating types were compared in a study with 26 participants who attempted the same five tasks with two software applications. The types were a Likert scale, a Usability Magnitude Estimation (UME) judgment, and a Subjective Mental Effort Question (SMEQ). All three types could distinguish between the applications with 26 participants, but the Likert and SMEQ types were more sensitive with small sample sizes. Both the Likert and SMEQ types were easy to learn and quick to execute. The online version of the SMEQ question was highly correlated with other measures and had equal sensitivity to the Likert question type.

This paper reports on the approach and main results of CUE-4, the fourth in a series of Comparative Usability Evaluation studies. A total of 17 experienced professional teams independently evaluated the usability of the website for the Hotel Pennsylvania in New York. Nine teams used usability testing while eight teams used expert reviews. The CUE-4 results document a wide difference in resources applied and issues reported. The teams reported 340 different usability issues. Only nine of these issues were reported by more than half of the teams, while 205 issues (60%) were uniquely reported, that is, no two teams reported the same issue. A total of 61 of the 205 uniquely reported issues were classified as serious or critical problems. The study also shows that there was no practical difference between the results obtained from usability testing and expert reviews for the issues identified. It was not possible to prove the existence of either missed problems or false alarms in expert reviews. The paper further discusses quality measures for usability evaluation productivity.