Perceived clutter is a potentially important but understudied construct in UX research. In this paper we described the development and assessment of a standardized questionnaire for reliable and valid measurement of perceived clutter of websites. Starting with an initial set of 16 items and two hypothesized factors, a series of exploratory analyses led to a final set of five items, two for the hypothesized construct of Content Clutter (too much irrelevant content like ads and videos) and three for the hypothesized construct of Design Clutter (poor design of relevant information like too much text, an unpleasant layout, or too much visual noise). Confirmatory analyses using an independent dataset showed excellent fit statistics for CFA of the five-item questionnaire and good fit for an SEM of the connections between clutter and other UX constructs. Researchers should exercise caution about generalizing results to other contexts and interfaces, but UX practitioners should be able to use this perceived clutter of websites (PCW) questionnaire when assessing consumer websites.

We replicated and extended previous research to investigate the extent to which perceived ease of use and perceived usefulness account for variation in overall experience, likelihood to recommend, intention to use, and reported usage in a three-month follow-up. Consistent with previous research, we found little effect on structural equation models from varying three measures of perceived ease and two measures of perceived usefulness. All models had statistically significant standardized estimates and squared multiple correlations and had acceptable fit statistics. Despite these manipulations, the models supported a consistent narrative. Both perceived ease and perceived usefulness are important antecedents that either directly or indirectly affect the experiential and intentional outcomes (perceived usefulness somewhat more than perceived ease), with intention to use accounting for 19% of variation in follow-up ratings of usage. These models support UX practitioners by demonstrating the importance of work that improves perceptions of product ease and usefulness and showing that the two-item UX-Lite questionnaire is an effective and efficient measure of perceived ease and usefulness.

In response to recent criticism of the usefulness of the construct of usability, we investigated the relationships between measures of perceived usability and the components of a modified version of the Technology Acceptance Model (mTAM) – Perceived Usefulness (PU) and Perceived Ease-of-Use (PEU). In three surveys, respondents used SUS, UMUX-LITE and mTAM to rate their actual (as opposed to expected) experience with three software products. As expected, the correlations between PEU and other measures of perceived usability tended to be significantly stronger than those with PU. Additional findings support the use of the UMUX-LITE as a compact measure of perceived usability that has a strong relationship to the mTAM and strong correspondence with concurrently collected SUS scores. The main theoretical result of this research were regression results providing evidence that the PEU component of the mTAM appears to be another measure of the construct of perceived usability, connecting the TAM to the construct of perceived usability through the mTAM and providing evidence against the claim that the construct of usability is a theoretical dead end.

This research continued previous investigation of the relationships among measures of perceived usability: the System Usability Scale (SUS), three metrics derived from the Usability Metric for User Experience (UMUX), and the Computer System Usability Questionnaire (CSUQ), this time with ratings of four everyday products (Excel, Word, Amazon, and Gmail). SUS ratings of these products were generally consistent with previous reports. Significant differences in SUS means across studies could be due to differences in frequency of use, with implications for using these data as usability benchmarks. Correspondence among the various measures of perceived usability was also consistent with previous research. Considering frequency of use, mean differences ranged from -2.0 to 1.8 (average shift in Sauro-Lewis grade range from -0.6 to 0.8). When SUS scores were above average, the range restriction of the UMUX-LITEr led to relatively large discrepancies with SUS, suggesting it might not always be better than unadjusted UMUXLITE.

Published with license by Taylor & Francis Group, LLC. This article aims to investigate the Grand Challenges which arise in the current and emerging landscape of rapid technological evolution towards more intelligent interactive technologies, coupled with increased and widened societal needs, as well as individual and collective expectations that HCI, as a discipline, is called upon to address. A perspective oriented to humane and social values is adopted, formulating the challenges in terms of the impact of emerging intelligent interactive technologies on human life both at the individual and societal levels. Seven Grand Challenges are identified and presented in this article: Human-Technology Symbiosis; Human-Environment Interactions; Ethics, Privacy and Security; Well-being, Health and Eudaimonia; Accessibility and Universal Access; Learning and Creativity; and Social Organization and Democracy. Although not exhaustive, they summarize the views and research priorities of an international interdisciplinary group of experts, reflecting different scientific perspectives, methodological approaches and application domains. Each identified Grand Challenge is analyzed in terms of: concept and problem definition; main research issues involved and state of the art; and associated emerging requirements. BACKGROUND This article presents the results of the collective effort of a group of 32 experts involved in the community of the Human Computer Interaction International (HCII) Conference series. The group’s collaboration started in early 2018 with the collection of opinions from all group members, each asked to independently list and describe five HCI grand challenges. During a one-day meeting held on the 20th July 2018 in the context of the HCI International 2018 Conference in Las Vegas, USA, the identified topics were debated and challenges were formulated in terms of the impact of emerging intelligent interactive technologies on human life both at the individual and societal levels. Further analysis and consolidation led to a set of seven Grand Challenges presented herein. This activity was organized and supported by the HCII Conference series.

The System Usability Scale (SUS) is the most widely used standardized questionnaire for the assessment of perceived usability. This review of the SUS covers its early history from inception in the 1980s through recent research and its future prospects. From relatively inauspicious beginnings, when its originator described it as a “quick and dirty usability scale,” it has proven to be quick but not “dirty.” It is likely that the SUS will continue to be a popular measurement of perceived usability for the foreseeable future. When researchers and practitioners need a measure of perceived usability, they should strongly consider using the SUS.

The primary purpose of this research was to investigate the relationship between two widely used questionnaires designed to measure perceived usability: the Computer System Usability Questionnaire (CSUQ) and the System Usability Scale (SUS). The correlation between concurrently collected CSUQ and SUS scores was 0.76 (over 50% shared variance). After converting CSUQ scores to a 0–100-point scale (to match the range of the SUS scores), there was a small but statistically significant difference between CSUQ and SUS means. Although this difference (just under 2 scale points out of a possible 100) was statistically significant, it did not appear to be practically significant. Although usability practitioners should be cautious pending additional independent replication, it appears that CSUQ scores, after conversion to a 0–100-point scale, can be interpreted with the Sauro–Lewis curved grading scale. As a secondary research goal, investigation of variations of the Usability Metric for User Experience (UMUX) replicated previous findings that the regression-adjusted version of the UMUX-LITE (UMUX-LITEr) had the closest correspondence with concurrently collected SUS scores. Thus, even though these three standardized questionnaires were independently developed and have different item content and formats, they largely appear to be measuring the same thing, presumably, perceived usability.

The System Usability Scale (SUS) is a widely adopted and studied questionnaire for usability evaluation. It is technology independent and has been used to evaluate the perceived usability of a broad range of products, including hardware, software, and websites. In this article we present a Slovene translation of the SUS (the SUS-SI) along with the procedure used in its translation and psychometric evaluation. The results indicated that the SUS-SI has properties similar to the English version. Slovene usability practitioners should be able to use the SUS-SI with confidence when conducting user research.

This special issue on usability and the user experience focuses on the application of psychometrics to the development and evaluation of standardized usability and user experience questionnaires. The six articles cover advances in the measurement of perceived usability (assessment of the instruments themselves and their application to measuring the perceived usability of mobile applications), attitude toward the Internet, need fulfillment, and the emotional consequences of interaction. Given the benefits of measurement standardization and the new research presented in this issue, the articles should be of interest to human–computer interaction researchers and practitioners.

This special issue focuses on the evolution of development and assessment methodologies related to usability and user experience. The five articles have a diverse range of topics, including comparison of moderated and unmoderated think-aloud usability sessions, a new usability inspection method based on concept mapping, analysis of the fitness of scrum (agile) and kanban (lean) development methodologies to incorporate user experience methodologies, exploration of the relation between expectations and user experience, and a case study describing difficulties encountered when assessing usability “in the wild.” These articles should prove to be of value to practitioners and researchers with an interest in the evolution of usability and user experience methodologies.

This article describes the psychometric properties of the Emotional Metric Outcomes (EMO) questionnaire and the System Usability Scale (SUS) using data collected as part of a large-sample unmoderated usability study (n = 471). The EMO is a concise multifactor standardized questionnaire that provides an assessment of transaction-driven personal and relationship emotional outcomes, both positive and negative. The SUS is a well-known standardized usability questionnaire designed to assess perceived usability. In previous research, psychometric evaluation using data from a series of online surveys showed that the EMO and its component scales had high reliability and concurrent validity with loyalty and overall experience metrics but did not find the expected four-factor structure. Previous structural analyses of the SUS have had mixed results. Analysis of the EMO data from the usability study revealed the expected four-factor structure. The factor structure of the SUS appeared to be driven by item tone. The estimated reliability of the SUS (.90) was consistent with previous estimates. The EMO and its subscales were also quite reliable, with the estimates of reliability for the various EMO scales ranging from .86 to .96. Regression analysis using SUS, EMO, and Effort as predictors revealed different key drivers for the outcome metrics of Satisfaction and Likelihood-to-Recommend. The key recommendations are to include the EMO as part of the battery of poststudy standardized questionnaires, along with the SUS (or similar questionnaire), but to be cautious in reporting SUS subscales such as Usable and Learnable.

The purpose of this research was to investigate various measurements of perceived usability, in particular, to assess (a) whether a regression formula developed previously to bring Usability Metric for User Experience LITE (UMUX-LITE) scores into correspondence with System Usability Scale (SUS) scores would continue to do so accurately with an independent set of data; (b) whether additional items covering concepts such as findability, reliability, responsiveness, perceived use by others, effectiveness, and visual appeal would be redundant with the construct of perceived usability or would align with other potential constructs; and (c) the dimensionality of the SUS as a function of self-reported frequency of use and expertise. Given the broad use of and emerging interpretative norms for the SUS, it was encouraging that the regression equation for the UMUX-LITE worked well with this independent set of data, although there is still a need to investigate its efficacy with a broader set of products and methods. Results from a series of principal components analyses indicated that most of the additional concepts, such as findability, familiarity, efficiency, control, and visual appeal covered the same statistical ground as the other more standard metrics for perceived usability. Two of the other items (Reliable and Responsive) made up a reliable construct named System Quality. None of the structural analyses of the SUS as a function of frequency of use or self-reported expertise produced the expected components, indicating the need for additional research in this area and a need to be cautious when using the Usable and Learnable components described in previous research.

The philosopher of science J. W. Grove (1989) once wrote, “There is, of course, nothing strange or scandalous about divisions of opinion among scientists. This is a condition for scientific progress” (p. 133). Over the past 30 years, usability, both as a practice and as an emerging science, has had its share of controversies. It has inherited some from its early roots in experimental psychology, measurement, and statistics. Others have emerged as the field of usability has matured and extended into user-centered design and user experience. In many ways, a field of inquiry is shaped by its controversies. This article reviews some of the persistent controversies in the field of usability, starting with their history, then assessing their current status from the perspective of a pragmatic practitioner. Put another way: Over the past three decades, what are some of the key lessons we have learned, and what remains to be learned? Some of the key lessons learned are:• When discussing usability, it is important to distinguish between the goals and practices of summative and formative usability.• There is compelling rational and empirical support for the practice of iterative formative usability testing—it appears to be effective in improving both objective and perceived usability.• When conducting usability studies, practitioners should use one of the currently available standardized usability questionnaires.• Because “magic number” rules of thumb for sample size requirements for usability tests are optimal only under very specific conditions, practitioners should use the tools that are available to guide sample size estimation rather than relying on “magic numbers.”

This article describes the development and psychometric evaluation of the Emotional Metric Outcomes (EMO) questionnaire—a new questionnaire designed to assess the emotional outcomes of interaction, especially the interaction of customers with service-provider personnel or software. The EMO is a concise multifactor standardized questionnaire that provides an assessment of transaction-driven personal and relationship emotional outcomes, both positive and negative. The primary purpose of the EMO is to move beyond traditional assessment of satisfaction to achieve a more effective measurement of customers’ emotional responses to products and processes. Psychometric evaluation showed that the EMO and its component scales had high reliability and concurrent validity with loyalty and overall experience metrics in a variety of measurement contexts. Concurrent measurement with the System Usability Scale (SUS) indicated that reported significant correlation of the SUS with likelihood-to-recommend ratings are probably primarily due to emotional rather than utilitarian aspects of the SUS.

In this paper we present the UMUX-LITE, a two-item questionnaire based on the Usability Metric for User Experience (UMUX) [6]. The UMUX-LITE items are This system's capabilities meet my requirements and This system is easy to use." Data from two independent surveys demonstrated adequate psychometric quality of the questionnaire. Estimates of reliability were .82 and .83 -- excellent for a two-item instrument. Concurrent validity was also high, with significant correlation with the SUS (.81, .81) and with likelihood-to-recommend (LTR) scores (.74, .73). The scores were sensitive to respondents' frequency-of-use. UMUX-LITE score means were slightly lower than those for the SUS, but easily adjusted using linear regression to match the SUS scores. Due to its parsimony (two items), reliability, validity, structural basis (usefulness and usability) and, after applying the corrective regression formula, its correspondence to SUS scores, the UMUX-LITE appears to be a promising alternative to the SUS when it is not desirable to use a 10-item instrument.

This article describes the development of a standardized computer system usability questionnaire for use with speakers of the Turkish language, the Turkish Computer System Usability Questionnaire, Short Version (T-CSUQ-SV). This new questionnaire, based on the English-language CSUQ, underwent careful translation and transformation through comprehensive psychometric evaluation. The results of the psychometric evaluation revealed an acceptable level of reliability, appropriate construct validity, and sensitivity to manipulation, indicating that Turkish usability practitioners should be able to use the T-CSUQ-SV with confidence when conducting user research.

In 2009, Bargas-Avila, Lötscher, Orsini and Opwis published The Intranet Satisfaction Questionnaire (ISQ). In their research, Bargas-Avila et al. (2009, Intranet satisfaction questionnaire: development and validation of a questionnaire to measure user satisfaction with the intranet. Comput. Hum. Behav., 25, 1241–1250.) identified an important aspect of HCI for which there was no existing standardized satisfaction questionnaire—the evaluation of corporate Intranets. To fill this void, they developed the ISQ after two rounds of careful, detailed item development and analysis. The resulting instrument has 13 items with two subscales: Content Quality and Intranet Usability, with a large-sample estimated reliability (coefficient alpha) of 0.89. They used the 13-item version of the ISQ to evaluate the Intranets of six companies, varying in size and sector. Despite some analytical weaknesses associated with its psychometric evaluation, it appears to work well for its intended purpose.

In 2010, Kraig Finstad published (in this journal) 'The Usability Metric for User Experience'—the UMUX. The UMUX is a standardized usability questionnaire designed to produce scores similar to the System Usability Scale (SUS), but with 4 rather than 10 items. The development of the questionnaire followed standard psychometric practice. Psychometric evaluation of the final version of the UMUX indicated acceptable levels of reliability (internal consistency), concurrent validity, and sensitivity. Critical review of this research suggests that its weakest element was the structural analysis, which concluded that the UMUX is unidimensional based on insufficient evidence. Mixed-tone item content and parallel analysis of the eigenvalues point to a possible two-factor structure. This weakness, however, is of more theoretical than practical importance, given the overall scale's apparent reliability, validity, and sensitivity.

In The Voice in the Machine, Roberto Pieraccini takes the reader on a journey through the history of speech technologies, from the speaking machine of von Kempelen in 1804 through the current uses ...

When designing questionnaires there is a tradition of including items with both positive and negative wording to minimize acquiescence and extreme response biases. Two disadvantages of this approach are respondents accidentally agreeing with negative items (mistakes) and researchers forgetting to reverse the scales (miscoding).

The distribution of task time data in usability studies is positively skewed. Practitioners who are aware of this positive skew tend to report the sample median. Monte Carlo simulations using data from 61 large-sample usability tasks showed that the sample median is a biased estimate of the population median. Using the geometric mean to estimate the center of the population will, on average, have 13% less error and 22% less bias than the sample median. Other estimates of the population center (trimmed, harmonic and Winsorized means) had worse performance than the sample median.

Correlations between prototypical usability metrics from 90 distinct usability tests were strong when measured at the task-level (r between .44 and .60). Using test-level satisfaction ratings instead of task-level ratings attenuated the correlations (r between .16 and .24). The method of aggregating data from a usability test had a significant effect on the magnitude of the resulting correlations. The results of principal components and factor analyses on the prototypical usability metrics provided evidence for an underlying construct of general usability with objective and subjective factors.

Factor analysis of Post Study System Usability Questionnaire (PSSUQ) data from 5 years of usability studies (with a heavy emphasis on speech dictation systems) indicated a 3-factor structure consistent with that initially described 10 years ago: factors for System Usefulness, Information Quality, and Interface Quality. Estimated reliabilities (ranging from .83-.96) were also consistent with earlier estimates. Analyses of variance indicated that variables such as the study, developer, stage of development, type of product, and type of evaluation significantly affected PSSUQ scores. Other variables, such as gender and completeness of responses to the questionnaire, did not. Norms derived from this data correlated strongly with norms derived from the original PSSUQ data. The similarity of psychometric properties between the original and this PSSUQ data, despite the passage of time and differences in the types of systems studied, provide evidence of significant generalizability for the questionnaire, supporting its use by practitioners for measuring participant satisfaction with the usability of tested systems.

Previous empirical research in subjective time estimation and applied work in auditory interface design imply that designers can use auditory stimuli during system processing to manipulate users' perception of its duration. Two experiments investigate the effect of system response time (SRT) duration and rate of change of an auditory waiting cue on participants' subjective time estimates and perceived affect. The results showed that perceived SRT duration and ratings of perceived anxiety, stress, and impatience increased as ticking rate increased. However, with a slow rate (2-sec ticking), participants underestimated the duration of SRT, but indicated a significant increase in negative affect as compared with silent conditions. These results suggest that interface designers may reduce the subjective duration and negative affective states of SRT through carefully chosen, slow tempo system processing tones. The results of this research also stress the importance of thoughtful, informed interface design that makes contact with the empirical literature of the cognitive sciences.

In this introduction to the special issue of the International Journal of Human-Computer Interaction, I discuss some current topics in usability evaluation and indicate how the contributions to the issue relate to these topics. The contributions cover a wide range of topics in usability evaluation, including a discussion of usability science, how to evaluate usability evaluation methods, the effect and control of certain biases in the selection of evaluative tasks, a lack of reliability in problem detection across evaluators, how to adjust estimates of problem-discovery rates computed from small samples, and the effects of perception of hedonic and ergonomic quality on user ratings of a product's appeal.

There are 2 excellent reasons to compute usability problem-discovery rates. First, an estimate of the problem-discovery rate is a key component for projecting the required sample size for a usability study. Second, practitioners can use this estimate to calculate the proportion of discovered problems for a given sample size. Unfortunately, small-sample estimates of the problem-discovery rate suffer from a serious overestimation bias. This bias can lead to serious underestimation of required sample sizes and serious overestimation of the proportion of discovered problems. This article contains descriptions and evaluations of a number of methods for adjusting small-sample estimates of the problem-discovery rate to compensate for this bias. A series of Monte Carlo simulations provided evidence that the average of a normalization procedure and Good-Turing (Jelinek, 1997; Manning & Schutze, 1999) discounting produces highly accurate estimates of usability problem-discovery rates from small sample sizes.

This article describes recent research in subjective usability measurement at IBM, focused on evaluating the psychometric properties of questionnaires designed for use in scenario‐based usability evaluation. The questionnaires address evaluation at both a global overall system level and at a more detailed scenario level. The primary goals of this article are to (a) discuss the psychometric characteristics of IBM questionnaires that measure user satisfaction with computer system usability, and (b) provide the questionnaires, with administration and scoring instructions. For scenario‐level measurement, the 3‐item After‐Scenario Questionnaire (ASQ) has excellent internal consistency, with coefficient alphas across a set of scenarios ranging from .90 to .96. For more global assessment, the Post‐Study System Usability Questionnaire (PSSUQ) also has excellent internal consistency, with an overall coefficient alpha of .97. Preliminary principal factor analysis of 48 PSSUQ questionnaires suggested the presence of three factors named, after varimax rotation, System Usefulness, Information Quality, and Interface Quality, with corresponding coefficient alphas of .96, .91, and .91. Evaluation of 377 PSSUQ questionnaires (modified to allow mailing to respondents in their offices and referred to as the Computer System Usability Questionnaire, or CSUQ) confirmed the structure of the preliminary principal factor analysis. Consequently, usability practitioners can use these questionnaires to help them measure users’ satisfaction with the usability of computer systems in the context of scenario‐based usability studies.

Researchers in human‐computer interaction (HCI) often use discrete multipoint scales (such as 5‐ or 7‐point scales) to measure user satisfaction and preference. Many knowledgeable authors state that the median is the appropriate measure of central tendency for such ordinal scales, although others challenge this assertion. This article introduces a new point of view, based on a human factors consideration. When decision makers read a usability report or attend a briefing, they may make decisions based on the magnitude of the difference between the measures of central tendency for key dependent variables. A major criterion that should affect the choice of presenting means or medians is the strength of the relationship between this difference and the observed significance levels of appropriate statistical tests. The results from two series of “real‐world” usability studies showed that the mean difference correlated more than the median difference with the observed significance levels (both parametric and nonparametric) for discrete multipoint scale data. Therefore, for these scales in this measurement context, the mean can be a better measure of central tendency than the median. The results also provided evidence that mean differences for 7‐point scales correlate more strongly with observed significance levels than those for 5‐point scales.