We present an experiment comparing performance of 20 novice evaluators of accessibility carrying out Web Content Accessibility Guidelines 2.0 conformance reviews working individually to performance obtained when they work in teams of two. They were asked to first carry out an individual assessment of a web page. Later on, they were matched randomly to constitute a group of two and they were asked to revise their initial assessment and to produce a group assessment of the same page. Results indicate that significant differences were found for sensitivity (inversely related to false negatives: +8%) and agreement (when measured in terms of the majority view: +10%). Members of groups exhibited strong agreement on the evaluation results among them and with the group outcome. Other measures of validity and reliability are not significantly affected by group work. Practical implications of these findings are that, for example, when it is important to reduce the false-negative rate, then employing a group of two people is more useful than having individuals carrying out the assessment. Openings for future research include further explorations of whether similar results hold for groups larger than two or what is the effect of mixing people with different accessibility background. When novice accessibility evaluators work in groups, their ability to identify all the true problems increases (by 8%). Likewise, reliability of group evaluations increases (by 10%). Individual or group evaluations can be considered as equivalent methods with respect to false positives (if differences up to 8% in correctness are tolerated). Individual or group evaluations can be considered as equivalent methods with respect to overall effectiveness (if differences up to 11% in F -measure are tolerated).

No abstract available.

The problem we tackle deals with testing responsiveness of Web applications across different user platforms, which are combinations of browsers, operating systems, devices and device orientation. Because of the variety of such environmental conditions, the behavior of the Web application (the application under test, AUT) can differ widely. Failure modes such as buttons not being visible, feedback boxes located in the wrong position, user interface (UI) components not behaving properly are very common and may affect only some of the platforms. Even when practitioners adopt services such as Browserstack that provide virtualization of user platforms, identification of such defects is very demanding, and as a consequence it is seldom performed in a continuous integration fashion. Unfortunately, layout requirements of UIs tend to change very frequently, and the result is that changes are often not tested well enough. What is needed is a software test automation solution. We tackled this problem by adopting a model-based approach, centered on models that represent the data manipulated by the UI and its behavior. A compiler reads these models, integrates model annotations providing details about UI implementation and about test oracles, and produces Java source code of a test harness. The test harness hides details about the Document Object Model (DOM), about the particular test driver that is used to drive the AUT, and details about the oracles that can be used to check if the AUT satisfies certain properties, such as those dealing with its layout. The test engineer can write high level code for jUnit/TestNG test cases, which can then be run on all the desired platforms, using Selenium as a driver.

The equality of access – accessibility – is difficult to quantify, define, or agree upon. Our previous work analysed the responses of web accessibility specialists in regard to a number of pre-defined definitions of accessibility. While uncovering much, this analysis did not allow us to quantify the communities’ understanding of the relationship accessibility has with other domains and assess how the community scopes accessibility. In this case, we asked over 300 people, with an interest in accessibility, to answer 33 questions surrounding the relationship between accessibility, user experience (UX), and usability; inclusion and exclusion; and evaluation, in an attempt to harmonise our understanding of web accessibility. We found that respondents think that accessibility and usability are highly related and also think that accessibility is applicable to everyone and not just people with disabilities. Respondents strongly agree that accessibility must be grounded on user-centred practices and that accessibility evaluation is more than just inspecting source code; however, they are divided as to whether training in ‘Web Content Accessibility Guidelines’ is necessary or not to assess accessibility. These perceptions are important for usability and UX professionals, developers of automated evaluation tools, and those practitioners running website evaluations.

Web accessibility means that disabled people can effectively perceive, understand, navigate, and interact with the web. Web accessibility evaluation methods are needed to validate the accessibility of web pages. However, the role of subjectivity and of expertise in such methods is unknown and has not previously been studied. This article investigates the effect of expertise in web accessibility evaluation methods by conducting a Barrier Walkthrough (BW) study with 19 expert and 57 nonexpert judges. The BW method is an evaluation method that can be used to manually assess the accessibility of web pages for different user groups such as motor impaired, low vision, blind, and mobile users. Our results show that expertise matters, and even though the effect of expertise varies depending on the metric used to measure quality, the level of expertise is an important factor in the quality of accessibility evaluation of web pages. In brief, when pages are evaluated with nonexperts, we observe a drop in validity and reliability. We also observe a negative monotonic relationship between number of judges and reproducibility: more evaluators mean more diverse outputs. After five experts, reproducibility stabilizes, but this is not the case with nonexperts. The ability to detect all the problems increases with the number of judges: With 3 experts all problems can be found, but for such a level 14 nonexperts are needed. Even though our data show that experts rated pages differently, the difference is quite small. Finally, compared to nonexperts, experts spent much less time and the variability among them is smaller, they were significantly more confident, and they rated themselves as being more productive. The article discusses practical implications regarding how BW results should be interpreted, how to recruit evaluators, and what happens when more than one evaluator is hired. Supplemental materials are available for this article. Go to the publisher's online edition of Human-Computer Interaction for statistical details and additional measures for this article. © 2011 Taylor & Francis Group, LLC.

The fact that several web accessibility metrics exist may be evidence of a lack of a comparison framework that highlights how well they work and for what purposes they are appropriate. In this paper we aim at formulating such a framework, demonstrating that it is feasible, and showing the findings we obtained when we applied it to seven existing automatic accessibility metrics. The framework encompasses validity, reliability, sensitivity, adequacy and complexity of metrics in the context of four scenarios where the metrics can be used. The experimental demonstration of the viability of the framework is based on applying seven published metrics to more than 1500 web pages and then operationalizing the notions of validity-as-conformance, adequacy and complexity. Our findings lead us to conclude that the Web Accessibility Quantitative Metric, Page Measure and Web Accessibility Barrier are the metrics that achieve the highest levels of quality (out of the seven that we examined). Finally, since we did not analyse reliability, sensitivity and validity-in-use, this paper provides guidance to address them in what are new research avenues.

World Wide Web accessibility and best practice audits and evaluations are becoming increasingly complicated, time consuming, and costly because of the increasing number of conformance criteria which need to be tested. In the case of web access by disabled users and mobile users, a number of commonalities have been identified in usage, which have been termed situationally-induced impairments; in effect the barriers experienced by mobile web users have been likened to those of visually disabled and motor impaired users. In this case, we became interested in understanding if it was possible to evaluate the problems of mobile web users in terms of the aggregation of barriers-to-access experienced by disabled users; and in this way attempt to reduce the need for the evaluation of the additional conformance criteria associated with mobile web best practice guidelines. We used the Barrier Walkthrough (BW) method as our analytical framework. Capable of being used to evaluate accessibility in both the disabled and mobile contexts, the BW method would also enable testing and aggregation of barriers across our target user groups. We tested 61 barriers across four user groups each over four pages with 19 experts and 57 non-experts focusing on the validity and reliability of our results. We found that 58% of the barrier types that were correctly found were identified as common between mobile and disabled users. Further, if our aggregated barriers alone were used to test for mobile conformance only four barrier types would be missed. Our results also showed that mobile users and low vision users have the most common barrier types, while low vision and motor impaired users experiencing similar rates of severity in the barriers they experienced. We conclude that the aggregated evaluation results for blind, low vision and motor impaired users can be used to approximate the evaluation results for mobile web users. © 2011 British Informatics Society Limited. Published by Elsevier B.V. All rights reserved.

Web Content Accessibility Guidelines 2.0 (WCAG 2.0) require that success criteria be tested by human inspection. Further, testability of WCAG 2.0 criteria is achieved if 80% of knowledgeable inspectors agree that the criteria has been met or not. In this paper we investigate the very core WCAG 2.0, being their ability to determine web content accessibility conformance. We conducted an empirical study to ascertain the testability of WCAG 2.0 success criteria when experts and non-experts evaluated four relatively complex web pages; and the differences between the two. Further, we discuss the validity of the evaluations generated by these inspectors and look at the differences in validity due to expertise.

This article reviews empirical research conducted in the last decade on the subject of how online display advertising affects the usability and quality of user experience of websites. In particular, from an in-depth analysis of research questions, methods, and findings of the reviewed studies, the following is discussed: (a) which conceptual and theoretical background knowledge, based on psychological explanations of user cognition, affection and behavior, can best support the design and investigation of online advertising, and (b) which specific adverts features and properties are key to understand and favor certain types of effects on users. By capitalizing on this benchmark knowledge on benefits of adverts and their hidden costs, web researchers and practitioners are encouraged to approach online advertising from a deeper and more comprehensive perspective, which is centered on qualities of web interaction that go beyond traditional usability factors. It is speculated that many of the theories and models developed for advertising effectiveness, and variables used to measure it, could and should be applied also when assessing the quality of the user experience when using websites in general, regardless of whether they contain adverts.

Although widely used,Web Content Accessibility Guidelines (WCAG) have not been studied from the viewpoint of their validity and reliability. WCAG 2.0 explicitly claim that they are based on "testable" criteria, but no scientific evidence exists that this is actually the case. Validity (how well all and only the true problems can be identified) and reliability (the extent to which different evaluations of the same page lead to same results) are key factors for quality of accessibility evaluation methods. They need to be well studied and understood for methods, and guidelines, that are expected to have a major impact.

Manual accessibility evaluation plays an important role in validating the accessibility of Web pages. This role has become increasingly critical with the advent of the Web Content Accessibility Guidelines (WCAG) 2.0 and their reliance on user evaluation to validate certain conformance measures. However, the role of expertise, in such evaluations, is unknown and has not previously been studied. This paper sets out to investigate the interplay between expert and non-expert evaluation by conducting a Barrier Walkthrough (BW) study with 19 expert and 51 non-expert judges. The BW method provides an evaluation framework that can be used to manually assess the accessibility of Web pages for different user groups including motor impaired, hearing impaired, low vision, cognitive impaired, etc. We conclude that the level of expertise is an important factor in the quality of accessibility evaluation of Web pages. Expert judges spent significantly less time than non-experts; rated themselves as more productive and confident than non-experts; and ranked and rated pages differently against each type of disability. Finally, both effectiveness and reliability of the expert judges are significantly higher than non-expert judges.

Accessibility auditors have to choose a method when evaluating accessibility: expert review (a.k.a. conformance testing), user testing, subjective evaluations, barrier walkthrough are some possibilities. However, little is known to date about their relative strengths and weaknesses. Furthermore, what happened for usability evaluation methods is likely to repeat for accessibility: that there is uncertainty about not only pros and cons of methods, but also about criteria to be used to compare them and metrics to measure these criteria. After a quick review and description of methods, the paper illustrates a comparative test of two web accessibility evaluation methods: conformance testing and barrier walkthrough. The comparison aims at determining merits of barrier walkthrough, using conformance testing as a control condition. A comparison framework is outlined, followed by the description of a laboratory experiment with 12 subjects (novice accessibility evaluators), and its results. Significant differences were found in terms of correctness, one of the several metrics used to compare the methods. Reliability also appears to be different.

Although they play an important role in any assessment procedure, web accessibility metrics are not yet well developed and studied. In addition, most metrics are geared towards conformance, and therefore are not well suited to answer questions whether the web site has critical barriers with respect to some user group.

Except for trivial cases, any accessibility evaluation has to be based on some method for selecting pages to be analyzed. But this selection process may bias the evaluation. Up to know, not much is known about available selection methods, and about their effectiveness and efficiency. The paper addresses the following open issues: how to define the quality of the selection process, which processes are better than others, how to measure their difference in quality, which factors may affect quality (type of assessment, size of the page pool, structural features of the web site). These issues are investigated through an experimental evaluation of thirteen sampling methods applied to 32000 web pages. While some of the conclusions are not surprising (for example, that sample size affect accuracy), others were not expected at all (that minimal sampling size obtains a high accuracy level under certain circumstances).

Abstract Although no unified definition of the concept of search strategy in Information Retrieval (IR) exists so far, its importance is manifest: nonexpert users, directly interacting with an IR system, apply a limited portfolio of simple actions; they do not know how to react in critical situations; and they often do not even realize that their difficulties are due to strategic problems. A user interface to an IR system should therefore provide some strategic help, focusing user's attention on strategic issues and providing tools to generate better strategies. Because neither the user nor the system can autonomously solve the information problem, but they complement each other, we propose a collaborative coaching approach, in which the two partners cooperate: the user retains the control of the session and the system provides suggestions. The effectiveness of the approach is demonstrated by a conceptual analysis, a prototype knowledge‐based system named FIRE, and its evaluation through informal laboratory experiments.

Article Evaluating user interfaces to information retrieval systems: a case study on user support Share on Authors: Giorgio Brajnik Dipartimento di Matematica e Informatica, University of Udine, Via delle Scienze, 206, Loc. Rizzi - 33100 Udine - ITALY Dipartimento di Matematica e Informatica, University of Udine, Via delle Scienze, 206, Loc. Rizzi - 33100 Udine - ITALYView Profile , Stefano Mizzaro Dipartimento di Matematica e Informatica, University of Udine, Via delle Scienze, 206, Loc. Rizzi - 33100 Udine - ITALY Dipartimento di Matematica e Informatica, University of Udine, Via delle Scienze, 206, Loc. Rizzi - 33100 Udine - ITALYView Profile , Carlo Tasso Dipartimento di Matematica e Informatica, University of Udine, Via delle Scienze, 206, Loc. Rizzi - 33100 Udine - ITALY Dipartimento di Matematica e Informatica, University of Udine, Via delle Scienze, 206, Loc. Rizzi - 33100 Udine - ITALYView Profile Authors Info & Claims SIGIR '96: Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrievalAugust 1996 Pages 128–136https://doi.org/10.1145/243199.243249Online:18 August 1996Publication History 46citation1,584DownloadsMetricsTotal Citations46Total Downloads1,584Last 12 Months16Last 6 weeks6 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

The requirements of a user modeling component for an expert interface are analyzed, and the main points of a proposed approach to user modeling are stated. The authors focus on a knowledge-based system, called UM-tool, devoted to creating, maintaining, and using explicit user models within an expert interface. UM-tool supports a novel approach to user modeling, which is based both on the use of stereotypes and on a dynamic reclassification scheme. The architecture of the system is described, the organization and content of its knowledge bases are illustrated, and the modeling mechanisms utilized are presented in detail. An example of the use of UM-tool in the frame of the information-retrieval-natural-language-interface (IR-NLI II) expert interface devoted to supporting end users in accessing online information retrieval services is discussed, focusing on the specific role of the user modeling component. An evaluation of the proposed approach and a critical comparison with related work are presented. Future research directions are outlined.>

This paper addresses the problem of building expert interfaces to information retrieval systems. In particular, the problem of augmenting the capabilities of such interfaces with user modeling features is discussed and the main benefits of this approach are outlined. The paper presents a prototype system called IR-NLI II, devoted to model by means of artificial intelligence techniques the human intermediary to information retrieval systems. The overall organization of the IR-NLI II system is presented, together with a short description of the two main modules implemented so far, namely the Information Retrieval Expert Subsystem and the User Modeling Subsystem. An example of interaction with IR-NLI II is described. Perspectives and future research directions are finally outlined.