Web browsing has never been easy for blind people, primarily due to the serial press-and-listen interaction mode of screen readers -- their "go-to'' assistive technology. Even simple navigational browsing actions on a page require a multitude of shortcuts. Auto-suggesting the next browsing action has the potential to assist blind users in swiftly completing various tasks with minimal effort. Extant auto-suggest feature in web pages is limited to filling form fields; in this paper, we generalize it to any web screen-reading browsing action, e.g., navigation, selection, etc. Towards that, we introduce SuggestOmatic, a personalized and scalable unsupervised approach for predicting the most likely next browsing action of the user, and proactively suggesting it to the user so that the user can avoid pressing a lot of shortcuts to complete that action. SuggestOmatic rests on two key ideas. First, it exploits the user's Action History to identify and suggest a small set of browsing actions that will, with high likelihood, contain an action which the user will want to do next, and the chosen action is executed automatically. Second, the Action History is represented as an abstract temporal sequence of operations over semantic web entities called Logical Segments - a collection of related HTML elements, e.g., widgets, search results, menus, forms, etc.; this semantics-based abstract representation of browsing actions in the Action History makes SuggestOmatic scalable across websites, i.e., actions recorded in one website can be used to make suggestions for other similar websites. We also describe an interface that uses an off-the-shelf physical Dial as an input device that enables SuggestOmatic to work with any screen reader. The results of a user study with 12 blind participants indicate that SuggestOmatic can significantly reduce the browsing task times by as much as 29% when compared with a hand-crafted macro-based web automation solution.

A screen reader's sequential press-and-listen interface makes for an unsatisfactory and often times painful web-browsing experience for blind people. To help alleviate this situation, we introduce Web Screen Reading Automation Assistant (SRAA) for automating users' screen-reading actions (e.g., finding price of an item) on demand, thereby letting them focus on what they want to do rather than on how to get it done. The key idea is to elevate the interaction from operating on (syntactic) HTML elements, as is done now, to operating on web entities (which are semantically meaningful collections of related HTML elements, e.g., search results, menus, widgets, etc.). SRAA realizes this idea of semantic abstraction by constructing a Web Entity Model (WEM), which is a collection of web entities of the underlying webpage, using an extensive generic library of custom-designed descriptions of commonly occurring web entities across websites. The WEM brings blind users closer to how sighted people perceive and operate on web entities, and together with a natural-language user interface, SRAA relieves users from having to press numerous shortcuts to operate on low-level HTML elements - the principal source of tedium and frustration. This paper describes the design and implementation of SRAA. Evaluation with 18 blind subjects demonstrates its usability and effectiveness.

Graphical user interfaces (GUIs) are widely used on smartphones, tablets, and laptops. While GUIs are convenient for sighted users, their accessibility for blind people, who use screen readers to interact with GUIs, remains to be problematic. Even the most screen-reader accessible GUIs are far less usable for blind people compared to sighted people, because the former group cannot benefit from the geometric layout of GUIs. As a result, blind people often have to listen through a lot of irrelevant content before they find what they are looking for. Haptic interfaces (those providing tactile feedback) have the potential to make GUI interfaces more accessible and usable for blind people. Alas, mainstream computer devices do not have haptic screens that would enable high-resolution tactile feedback, and specialized haptic devices are very limited and/or are exuberantly expensive and bulky.

Web browsing with screen readers is tedious and frustrating, largely due to the inability of blind screen-reader users to get spatial information about the structure of web pages and utilize it for effective navigation. Haptic interfaces have the potential to provide blind users with a tactile --feel-- for the 2-D layout of web pages and help them focus screen reading on specific parts of the webpage. In this preliminary work, we explore the utility of a simple haptic web-browsing interface -- tactile overlays, and report on a preliminary user study with 10 blind participants who performed various web-browsing tasks with and without these overlays. We also analyzed the user-interaction behavior and explored the appropriate design choices and their tradeoffs in the space of haptic-interface design for accessible web browsing

Speech-enabled dialogue systems have the potential to enhance the ease with which blind individuals can interact with the Web beyond what is possible with screen read-ers- the currently available assistive tech-nology which narrates the textual content on the screen and provides shortcuts to navigate the content. In this paper, we present a dialogue act model towards de-veloping a speech enabled browsing sys-tem. The model is based on the corpus data that was collected in a wizard-of-oz study with 24 blind individuals who were assigned a gamut of browsing tasks. The development of the model included exten-sive experiments with assorted feature sets and classifiers; the outcomes of the exper-iments and the analysis of the results are presented. 1

While reading on touch-screens, sighted users can quickly pan through content, skim it, and pick out bits and pieces of information before deciding to read it more carefully. In contrast, blind users have to rely on the screen reader to narrate the content to them. To go through the text quickly, blind users employ gestures that direct the screen reader to skip to the next line or the next paragraph. However, the serial audio interface of the screen reader makes it difficult for blind users to get a sense of what is important before listening to, at least, a part of the content. This makes ad hoc skimming with gestures slow and ineffective. We address this problem in this paper; specifically we propose a non-visual skimming interface that enables blind users to control the amount of content with simple pinch-in and pinch-out gestures. This interface simulates the skimming experience enjoyed by sighted people, and enables blind users to listen to the gist of content, while controlling the speed of information intake. We report on a user study demonstrating that the proposed interface significantly outperforms ad hoc skimming techniques employed by blind users. Our results suggest that the proposed approach holds promise in empowering blind users to access digitized information much faster.

The Web is far less usable and accessible for people with vision impairments than it is for sighted people. Web automation, a process of automating browsing actions on behalf of the user, has the potential to bridge the divide between the ways sighted and people with vision impairment access the Web; specifically, it can enable the latter to breeze through web browsing tasks that beforehand were slow, hard, or even impossible to accomplish. Typical web automation requires that the user record a macro, a sequence of browsing steps, so that these steps can be automated in the future by replaying the macro. However, for people with vision impairment, automation with macros is not usable.

People with visual disabilities, especially those who are blind, have digital content narrated to them by text-to-speech (TTS) engines (e.g., with the help of screen readers). Naively narrating web pages, particularly the ones consisting of several diverse pieces (e.g., news summaries, opinion pieces, taxonomy, ads), with TTS engines without organizing them into thematic segments will make it very difficult for the blind user to mentally separate out and comprehend the essential elements in a segment, and the effort to do so can cause significant cognitive stress. One can alleviate this difficulty by segmenting web pages into thematic pieces and then narrating each of them separately. Extant segmentation methods typically segment web pages using visual and structural cues. The use of such cues without taking into account the semantics of the content, tends to produce "impure" segments containing extraneous material interspersed with the essential elements. In this paper, we describe a new technique for identifying thematic segments by tightly coupling visual, structural, and linguistic features present in the content. A notable aspect of the technique is that it produces segments with very little irrelevant content. Another interesting aspect is that the clutter-free main content of a web page, that is produced by the Readability tool and the "Reader" feature of the Safari browser, emerges as a special case of the thematic segments created by our technique. We provide experimental evidence of the effectiveness of our technique in reducing clutter. We also describe a user study with 23 blind subjects of its impact on web accessibility.

In our information-driven web-based society, we are all gradually falling ""victims"" to information overload [5].However, while sighted people are finding ways to sift through information faster, Internet users who are blind are experiencing an even greater information overload. These people access computers and Internet using screen-reader software, which reads the information on a computer screen sequentially using computer-generated speech. While sighted people can learn how to quickly glance over the headlines and news articles online to get the gist of information, people who are blind have to use keyboard shortcuts to listen through the content narrated by a serial audio interface. This interface does not give them an opportunity to know what content to skip and what to listen to. So, they either listen to all of the content or listen to the first part of each sentence or paragraph before they skip to the next one. In this paper, we propose an automated approach to facilitate non-visual skimming of web pages. We describe the underlying algorithm, outline a non-visual skimming interface, and report on the results of automated experiments, as well as on our user study with 23 screen-reader users. The results of the experiments suggest that we have been moderately successful in designing a viable algorithm for automatic summarization that could be used for non-visual skimming. In our user studies, we confirmed that people who are blind could read and search through online articles faster and were able to understand and remember most of what they have read with our skimming system. Finally, all 23 participants expressed genuine interest in using non-visual skimming in the future.

People with disabilities have always overcome accessibility problems by enlisting people in their community to help. The Internet has broadened the available community and made it easier to get on-demand assistance remotely. In particular, the past few years have seen the development of technology in both research and industry that uses human power to overcome technical problems too difficult to solve automatically. In this paper, we frame recent developments in human computation in the historical context of accessibility, and outline a framework for discussing new advances in human-powered access technology. Specifically, we present a set of 13 design principles for human-powered access technology motivated both by historical context and current technological developments. We then demonstrate the utility of these principles by using them to compare several existing human-powered access technologies. The power of identifying the 13 principles is that they will inspire new ways of thinking about human-powered access technologies.

People who are blind use screen readers for browsing web pages. Since screen readers read out content serially, a naive readout tends to mix irrelevant and relevant content thereby disrupting the coherency of the material being read out and confusing the listener. To address this problem we can partition web pages into coherent segments and narrate each such piece separately. Extant methods to do segmentation use visual and structural cues without taking the semantics into account and consequently create segments containing irrelevant material. In this paper, we describe a new technique for creating coherent segments by tightly coupling visual, structural, and linguistic features present in the content. A notable aspect of the technique is that it produces segments with little irrelevant content. Preliminary experiments indicate that the technique is effective in creating highly coherent segments and the experiences of an early adopter who is blind suggest that it enriches the overall browsing experience.

This demonstration will propose a touch-based directional navigation technique, on touch interface (e.g., iPhone, Macbook) for people with visual disabilities especially blind individuals. Such interfaces coupled with TTS (text-to-speech) systems open up intriguing possibilities for browsing and skimming web content with ease and speed. Apple's seminal VoiceOver system for iOS is an exemplar of bringing touch-based web navigation to blind people. There are two major shortcomings: "fat finger" and "finger-fatigue" problems, which have been addressed in this paper with two proposed approaches. A preliminary user evaluation of the system incorporating these ideas suggests that they can be effective in practice.

Advances in web technology have considerably widened the Web accessibility divide between sighted and blind users. This divide is especially acute when conducting online transactions, e.g., shopping, paying bills, making travel plans, etc. Such transactions span multiple web pages and require that users find clickable objects (e.g., “add-to-cart” button) which are essential for transaction progress. While this is fast and straightforward for sighted users, locating the clickable objects causes considerable strain for blind individuals using screen-reading technology. Screen readers force users to listen to irrelevant information sequentially and provide no interface for identifying relevant clickable objects. This paper addresses the problem of making clickable objects readily accessible, which can substantially reduce the information overload that is otherwise experienced by blind users. A static knowledge base of keywords constructed from the captions of clickable objects does not provide enough learning capability for identifying clickable objects which do not have any captions (e.g., image buttons without alternative text). In this paper, we present an Information Retrieval based technique that uses the context of transaction-centric objects (e.g., “add-to-cart” and “checkout” buttons) to identify and classify them even when their captions are missing. In addition, the technique utilizes a reinforcement mechanism based on user feedback to accommodate previously unseen captions of objects as well as new categories of objects. We provide user study and experimental evidence of the effectiveness of our algorithm.

An important aspect of making the Web accessible to blind users is ensuring that all important web page elements such as links, clickable buttons, and form fields have explicitly assigned labels. Properly labeled content is then correctly read out by screen readers, a dominant assistive technology used by blind users. In particular, improperly labeled form fields can critically impede online transactions such as shopping, paying bills, etc. with screen readers. Very often labels are not associated with form fields or are missing altogether, making form filling a challenge for blind users. Algorithms for associating a form element with one of several candidate labels in its vicinity must cope with the variability of the element's features including label's location relative to the element, distance to the element, etc. Probabilistic models provide a natural machinery to reason with such uncertainties. In this paper we present a Finite Mixture Model (FMM) formulation of the label association problem. The variability of feature values are captured in the FMM by a mixture of random variables that are drawn from parameterized distributions. Then, the most likely label to be paired with a form element is computed by maximizing the log-likelihood of the feature data using the Expectation-Maximization algorithm. We also adapt the FMM approach for two related problems: assigning labels (from an external Knowledge Base) to form elements that have no candidate labels in their vicinity and for quickly identifying clickable elements such as add-to-cart, checkout, etc., used in online transactions even when these elements do not have textual captions (e.g., image buttons w/o alternative text). We provide a quantitative evaluation of our techniques, as well as a user study with two blind subjects who used an aural web browser implementing our approach.

This demo will present HearSay, a multi-modal non-visual web browser, which aims to bridge the growing Web Accessibility divide between individuals with visual impairments and their sighted counterparts, and to facilitate full participation of blind individuals in the growing Web-based society.

Screen readers, the dominant assistive technology used by visually impaired people to access the Web, function by speaking out the content of the screen serially. Using screen readers for conducting online transactions can cause considerable information overload, because transactions, such as shopping and paying bills, typically involve a number of steps spanning several web pages. One can combat this overload by using a transaction model for web accessibility that presents only fragments of web pages that are needed for doing transactions. We can realize such a model by coupling a process automaton, encoding states of a transaction, with concept classifiers that identify page fragments “relevant ” to a particular state of the transaction. In this paper we present a fully automated process that synergistically combines several techniques for transforming unlabeled

Non-visual aural web browsing remains inefficient as compared to regular browsing with visual modalities. This paper proposes an approach for automation of repetitive browsing tasks by using personalized macros, which are easy to record and replay with a speech-enabled interface. The prototype system is implemented in the framework of the HearSay non-visual web browser.

Web applications facilitated by technologies such as JavaScript, DHTML, AJAX, and Flash use a considerable amount of dynamic web content that is either inaccessible or unusable by blind people. Server side changes to web content cause whole page refreshes, but only small sections of the page update, causing blind web users to search linearly through the page to find new content. The connecting theme is the need to quickly and unobtrusively identify the segments of a web page that have changed and notify the user of them. In this paper we propose Dynamo, a system designed to unify different types of dynamic content and make dynamic content accessible to blind web users. Dynamo treats web page updates uniformly and its methods encompass both web updates enabled through dynamic content and scripting, and updates resulting from static page refreshes, form submissions, and template-based web sites. From an algorithmic and interaction perspective Dynamo detects underlying changes and provides users with a single and intuitive interface for reviewing the changes that have occurred. We report on the quantitative and qualitative results of an evaluation conducted with blind users. These results suggest that Dynamo makes access to dynamic content faster, and that blind web users like it better than existing interfaces.

Research projects, assistive technology, and individuals all create metadata in order to improve Web accessibility for visually impaired users. However, since these projects are disconnected from one another, this metadata is isolated in separate tools, stored in disparate repositories, and represented in incompatible formats. Web accessibility could be greatly improved if these individual contributions were merged. An integration method will serve as the bridge between future academic research projects and end users, enabling new technologies to reach end users more quickly. Therefore we introduce Accessibility Commons, a common infrastructure to integrate, store, and share metadata designed to improve Web accessibility. We explore existing tools to show how the metadata that they produce could be integrated into this common infrastructure, we present the design decisions made in order to help ensure that our common repository will remain relevant in the future as new metadata is developed, and we discuss how the common infrastructure component facilitates our broader social approach to improving accessibility.

People with visual impairments use screen readers to browse the Web. Sequential processing of web pages by screen readers causes information overload, making web browsing time-consuming and strenuous. These problems are further exacerbated in web transactions (e.g.: online shopping), which involve multiple steps spanning several web pages. In this paper we present a lightweight approach for doing Web transactions using non-visual modalities. We describe how analysis of context surrounding the link coupled with a shallow knowledge-base with patterns and keywords can help identify various concepts (e.g.: "add to cart", "item description", etc.) that are important in web transactions. Our preliminary results show promise that presenting such concepts to the users can reduce information overload and improve their overall browsing experience.

In this paper, we describe a general-purpose approach for partitioning Web page content. The novelty of our ap- proach lies in the use of detailed layout information from a Web page renderer to determine spatial locality and identify visual separators, and the use of relaxed matching over pre- sentation style information to determine presentation style similarity. We present several examples to illustrate the gen- erality of our approach.

Web sites are designed for graphical mode of interaction. Sighted users can visually segment Web pages and quickly identify relevant information. In contrast, visually-disabled individuals have to use screen readers to browse the Web. Screen readers process pages sequentially and read through everything, making Web browsing time-consuming and strenuous. The use of shortcut keys and searching offers some improvements, but the problem still remains. In this paper, we address this problem using the notion of context. When a user follows a link, we capture the context of the link, and use it to identify relevant information on the next page. The content of this page is rearranged, so that the relevant information is read out first. We conducted a series experiments to compare the performance of our prototype system with the state-of-the-art JAWS screen reader. Our results show that the use of context can potentially save browsing time as well as improve browsing experience of visually disabled individuals.

Except for a handful of "mobile" Web sites, the Web is designed for browsing using personal computers with large screens capable of fully rendering the content of most Web pages. Browsing with handhelds, such as small-screen PDA's or cell phones, usually involves a lot of horizontal and vertical scrolling, thus making Web browsing time-consuming and strenuous. At the same time, one isoften only interested in a fragment of a Web page, which again may not fit on the limited-size screens of mobile devices, requiring more scrolling in both dimensions. In this paper, we address the problem of browsing fatigue during mobile Web access using geometric segmentation of Web pages and the notion of context. Our prototype system, CMo, reduces information overload by allowing its users to see the most relevant fragment of the page and then navigate between other fragments if necessary. On following a link, CMo captures the context of the link, employing a simple topic-boundary detection technique; then, it uses the context to identify relevant information in the next page with the help of a Support Vector Machine, a statistical machine-learning model. Our experiments show that the use of context can potentially save browsing time and improve the mobile browsing experience.

Web sites are designed for graphical mode of interaction. Sighted users can "cut to the chase" and quickly identify relevant information in Web pages. On the contrary, individuals with visual disabilities have to use screen-readers tobrowse the Web. As screen-readers process pages sequentially and read through everything, Web browsing can become strenuous and time-consuming. Although, the use ofshortcuts and searching offers some improvements, the problem still remains. In this paper, we address the problemof information overload in non-visual Web access using thenotion of context. Our prototype system, CSurf, embodyingour approach, provides the usual features of a screen-reader.However, when a user follows a link, CSurf captures thecontext of the link using a simple topic-boundary detectiontechnique, and uses it to identify relevant information onthe next page with the help of a Support Vector Machine, astatistical machine-learning model. Then, CSurf reads the Web page starting from the most relevant section, identifiedby the model. We conducted a series experiments to evaluate the performance of CSurf against the state-of-the-artscreen-reader, JAWS. Our results show that the use of context can potentially save browsing time and substantiallyimprove browsing experience of visually disabled people.

VoiceXML (VXML) is a W3C's standard for specifying interactive dialogs. It finds multiple uses in variousWeb applications. VXML can also be used in non-visual Web browsing. There is no suitable, complete, open-source, flexible VXML interpreter to process VXML dialogs. My project is focusing on developing a VXML interpreter, VXMLSurf, that will be fully compliant with VXML 2.0 specifications and geared toward accessing Web content. VXMLSurf implements a number of extended features that provide blind users with more control over interactive browsing dialogs. VXMLSurf is a part of the HearSay project for developing a non-visual Web browser. The goal of the project is to make the Web more accessible for blind people.

To browse the Web, blind people have to use screen readers, which process pages sequentially, making browsing timeconsuming. We present a prototype system, CSurf, which provides all features of a regular screen reader, but when a user follows a link, CSurf captures the context of the link and uses it to identify relevant information on the next page. CSurf rearranges the content of the next page, so, that the relevant information is read out first. A series experiments have been conducted to evaluate the performance of CSurf.