Studies show Visual Question Answering (VQA) systems are valuable tools for people with visual impairments to quickly obtain information from an image. In this poster, we present our ongoing work towards developing uses of live photos to mitigate quality issues in photography from people with visual impairments for VQA. New contributions building on our prior research include an expanded live photos dataset, a more in-depth analysis of VQA results, and an analysis of features of our live photos compared with existing data collected from people with visual impairments. We show that live photos are a promising method for improving accuracy of VQA and that our sample live photos mimic the types of images taken in real-world settings by people with visual impairments for the task of VQA.

We present a multimodal deep learning framework that can generate summarization text supporting the main idea of an information graphic for presentation to a person who is blind or visually impaired. The framework utilizes the visual, textual, positional, and size characteristics extracted from the image to create the summary. Different and complimentary neural architectures are optimized for each task using crowdsourced training data. From our quantitative experiments and results, we explain the reasoning behind our framework and show the effectiveness of our models. Our qualitative results showcase text generated from our framework and show that Mechanical Turk participants favor them to other automatic and human generated summarizations. We describe the design and results of an experiment to evaluate the utility of our system for people who have visual impairments in the context of understanding Twitter Tweets containing line graphs.

Information graphics, e.g. line or bar graphs, are often displayed in documents and popular media to support an intended message, but for a growing number of people, they are missing the point. The World Health Organization estimates that the number of people with vision impairment could triple in the next thirty years due to population growth and aging. If a graphic is not described, explained in the text, or missing alt tags and other metadata (as is often the case in popular media), the intended message is lost or not adequately conveyed. In this work, we describe a multimodal deep learning approach that supports the communication of the intended message. The multimodal model uses both the pixel data and text data in a single neural network to classify the information graphic into an intention category that has previously been validated as useful for people who are blind or who are visually impaired. Furthermore, we collect a new dataset of information graphics and present qualitative and quantitative results that show our multimodal model exceeds the performance of any one modality alone, and even surpasses the capabilities of the average human annotator.

Currently, there is a lack of text complexity awareness in NLG systems.Much attention has been given to text simplification.However, based upon results of an experiment, we unveiled that sophisticated readers in fact would rather read more sophisticated text, instead of the simplest text they could get.Therefore, we propose a technique that considers different readability levels during the micro planning phase of an NLG system.Our technique considers grammatical and syntactic choices, as well as lexical items, when generating text.The application uses the domain of descriptive summaries of line graphs as its use case.The technique proposed uses learning for identifying features of text complexity; a graph search algorithm for efficient aggregation given a target reading level, and a combination of language modeling and word vectors for the creation of a domain-aware synset which allows the creation of disambiguated lexicon that is appropriate to different reading levels.We found that generating text at different target reading levels is indeed preferred by readers with varying reading abilities.To the best of our knowledge, this is the first time readability awareness is considered in the micro planning phase of NLG systems.

This paper presents the methodology for generating textual summaries of line graphs in the SIGHT (Summarizing Information GrapHics Textually) system and the evaluation of line graph summaries produced by SIGHT. The system is designed to deliver the high-level knowledge conveyed by informational graphics present in online popular media articles (newspaper and magazines) to individuals who do not have visual access to the image. It works by producing and delivering a concise summary of the graph's content including the most important visual features present in the graphic. The system is briefly described; the evaluation compares the utility of the generated textual summaries to visually viewing the graphic in order to answer important questions about the line graph.

We use efficient screening experiments to investigate and improve topic analysis based multi-document extractive summa-rization. In our summarization process, topic analysis determines the weighted topic content vectors that characterize the corpora, and then Jensen-Shannon diver-gence extracts sentences that best match the weighted content vectors to assemble the summaries. We use screening experi-ments to investigate several control param-eters in this process, gaining better under-standing of and improving the topic anal-ysis based summarization process. 1

Deciding on the complexity of a generated text in NLG systems is a contentious task.Some systems propose the generation of simple text for low-skilled readers; some choose what they anticipate to be a "good measure" of complexity by balancing sentence length and number of sentences (using scales such as the D-level sentence complexity) for the text; while others target high-skilled readers.In this work, we discuss an approach that aims to leverage the experience of the reader when reading generated text by matching the syntactic complexity of the generated text to the reading level of the surrounding text.We propose an approach for sentence aggregation and lexical choice that allows generated summaries of line graphs in multimodal articles available online to match the reading level of the text of the article in which the graphs appear.The technique is developed in the context of the SIGHT (Summarizing Information Graphics Textually) system.This paper tackles the micro planning phase of sentence generation discussing additionally the steps of lexical choice, and pronominalization.

This demo presents a Natural Language Gener-ation (NLG) system that generates summaries of informational graphics, specifically simple line graphs, present in popular media. The sys-tem is intended to capture the high-level knowledge conveyed by the graphic and its out-standing visual features. It comprises a content selection phase that extracts the most important content of the graphic, an organization phase, which orders the propositions in a coherent manner, and a realization phase that uses the text surrounding the article to make decisions on the choice of lexical items and amount of ag-gregation applied to the propositions to gener-ate the summary of the graphic. 1

Information retrieval research has made significant progress in the retrieval of text documents and images. However, relatively little attention has been given to the retrieval of information graphics (non-pictorial images such as bar charts and line graphs) despite their proliferation in popular media such as newspapers and magazines. Our goal is to build a system for retrieving bar charts and line graphs that reasons about the content of the graphic itself in deciding its relevance to the user query. This paper presents the first steps toward such a system, with a focus on identifying the category of intended message of potentially relevant bar charts and line graphs. Our learned model achieves accuracy higher than 80\% on a corpus of collected user queries.

When scanning through a text document for the answer to a question, visual readers are able to quickly locate text within the document related to the answer while simultaneously getting a general sense of the document's content. For nonvisual readers, however, this poses a challenge, especially when the relevant text is spread out or worded in a way that can't be searched for directly. Our goal is to make the scanning experience quicker for nonvisual readers by giving them an experience similar to that of visual readers. To do this we first determined what visual scanners focused on by using an eye-tracker while they scanned for answers to complex questions. Resulting data revealed that text with loose semantic connections to the question are important. This paper reports on our efforts to develop a method that automatically replicates the connections made by visual scanners. Ultimately, our goal is a system that replicates the visual scanning experience, allowing nonvisual readers to quickly glean information in a manner similar to how visual readers glean information when scanning. This work stems from work with students who are nonvisual readers and is aimed at making their school experience more equitable with students who scan visually.

Information graphics (such as bar charts and line graphs) play a vital role in many multimodal documents. The majority of information graphics that appear in popular media are intended to convey a message and the graphic designer uses deliberate communicative signals, such as highlighting certain aspects of the graphic, in order to bring that message out. The graphic, whose communicative goal (intended message) is often not captured by the document's accompanying text, contributes to the overall purpose of the document and cannot be ignored. This article presents our approach to providing the high-level content of a non-scientific information graphic via a brief textual summary which includes the intended message and the salient features of the graphic. This work brings together insights obtained from empirical studies in order to determine what should be contained in the summaries of this form of non-linguistic input data, and how the information required for realizing the selected content can be extracted from the visual image and the textual components of the graphic. This work also presents a novel bottom–up generation approach to simultaneously construct the discourse and sentence structures of textual summaries by leveraging different discourse related considerations such as the syntactic complexity of realized sentences and clause embeddings. The effectiveness of our work was validated by different evaluation studies.

Although intelligent interactive systems have been the focus of many research efforts, very few have addressed systems for individuals with disabilities. This article presents our methodology for an intelligent interactive system that provides individuals with sight impairments with access to the content of information graphics (such as bar charts and line graphs) in popular media. The article describes the methodology underlying the system's intelligent behavior, its interface for interacting with users, examples processed by the implemented system, and evaluation studies both of the methodology and the effectiveness of the overall system. This research advances universal access to electronic documents.

Interactive SIGHT is a system that is intended to provide people with visual impairments access to the kind of information graphics found in popular media (i.e., electronic newspapers or magazines). The majority of such graphics are intended to convey a message; the graphic designer chose the graphic and its design in order to make a point. Interactive SIGHT, which is implemented as a browser extension that works on simple bar charts, provides a brief high-level summary of the graphic through natural language text that is conveyed to the user as speech. The user may request further information about the graphic through a follow-up question facility which allows many follow-up responses to be generated. The demo will illustrate the system's methodology on several bar charts that have been preloaded along with some accompanying text.

This poster describes an approach to creating a tool to aid blind, low-vision, dyslexic, and other non-visual readers in skimming a document to answer questions. The goal is to give them information similar to that obtained by a visual reader's skimming experience. Towards this goal, we examine approaches to locating areas of interest within a document that are related to a question. The areas of interest in relation to a question were determined through user studies in which visual readers skimmed through documents for answers to questions while being tracked by an eye tracking system. We then examined methods of automatically identifying the areas of interest using keywords related to the content of the question. This poster focuses on the effectiveness of these different methods in identifying the areas of interest.

This poster describes an approach to creating a system that will allow blind, low vision, dyslexic, and other non-visual readers to skim through documents for answers to questions. The ultimate goal is a system that gives the users information similar to that which a sighted individual obtains while skimming through a document for an answer to a question. To create this system, we will be incorporating open domain question answering (QA) techniques, word clustering techniques, and data gathered from visual readers skimming through documents. Ultimately the system will take a question and a text document and return a list of rated links that suggest where the answers within a document are likely to be located. This poster focuses on results of data gathered from individuals skimming through documents.

Word prediction can be used to enhance the communication rate of people with disabilities who use Augmentative and Alternative Communication (AAC) devices. We use statistical methods in a word prediction system, which are trained on a corpus, and then measure the efficacy of the resulting system by calculating the theoretical keystroke savings on some held out data. Ideally training and testing should be done on a large corpus of AAC text covering a variety of topics, but no such corpus exists. We discuss training and testing on a wide variety of corpora meant to approximate text from AAC users. We show that training on a combination of in-domain data with out-of-domain data is often more beneficial than either data set alone and that advanced language modeling such as topic modeling is portable even when applied to very different text.

Word prediction can be used for enhancing the communication ability of persons with speech and language impairments. In this work, we explore two methods of adapting a language model to the topic of conversation, and apply these methods to the prediction of fringe words.

While automatic text summarization is an area that has received a great deal of attention in recent research, the problem of efficiency in this task has not been frequently addressed. When the size and quantity of documents available on the Internet and from other sources are considered, the need for a highly efficient tool that produces usable summaries is clear. We present a linear-time algorithm for lexical chain computation. The algorithm makes lexical chains a computationally feasible candidate as an intermediate representation for automatic text summarization. A method for evaluating lexical chains as an intermediate step in summarization is also presented and carried out. Such an evaluation was heretofore not possible because of the computational complexity of previous lexical chains algorithms.

In this paper we discuss our approach toward establishing a model of the acquisition of English grammatical structures by users of our English language tutoring system, which has been designed for deaf users of American Sign Language. We explore the correlation between a corpus of error-tagged texts and their holistic proficiency scores assigned by experts in order to draw initial conclusions about what language errors typically occur at different levels of proficiency in this population. Since errors made at lower levels (and not at higher levels) presumably represent constructions acquired before those on which errors are found only at higher levels, this should provide insight into the order of acquisition of English grammatical forms.

This paper describes progress toward a prototype implementation of a tool which aims to improve literacy in deaf high school and college students who are native (or near native) signers of American Sign Language (ASL). We envision a system that will take a piece of text written by a deaf student, analyze that text for grammatical errors, and engage that student in a tutorial dialogue, enabling the student to generate appropriate corrections to the text. A strong focus of this work is to develop a system which adapts this process to the knowledge level and learning strengths of the user and which has the flexibility to engage in multi-modal, multi-lingual tutorial instruction utilizing both English and the native language of the user.

We present a system which uses lexical chains as an intermediate representation for automatic text summarization. This system builds on previous research by implementing a lexical chain extraction algorithm in linear time. The system is reasonably domain independent and takes as input any text or HTML document. The system outputs a short summary based on the most salient concepts from the original document. The length of the extracted summary can be either controlled automatically, or manually based on length or percentage of compression. While still under development, the system provides useful summaries which compare well in information content to human generated summaries. Additionally, the system provides a robust test bed for future summary generation research.

The rapid growth of the Internet has resulted in enormous amounts of information that has become more difficult to access efficiently. Internet users require tools to help manage this vast quantity of information. The primary goal of this research is to create an efficient and effective tool that is able to summarize large documents quickly. This research presents a linear time algorithm for calculating lexical chains which is a method of capturing the “aboutness” of a document. This method is compared to previous, less efficient methods of lexical chain extraction. We also provide alternative methods for extracting and scoring lexical chains. We show that our method provides similar results to previous research, but is substantially more efficient. This efficiency is necessary in Internet search applications where many large documents may need to be summarized at once, and where the response time to the end user is extremely important.

Augmentative and Alternative Communication (AAC) is the field of study concernedwith providing devices and techniques to augment the communicative ability of a person whose disability makes it difficult to speak in an understandable fashion.For several years, we have been applying natural language processing techniques to the field of AAC in order to develop intelligent communication aids that attempt to provide linguistically "correct" output while speeding communication rate.In this paper we describe some of the interface issues that must be considered when developing such a device.We focus on a project aimed at a group of users who have cognitive impairments that affect their linguistic ability.A prototype system is under development which will hopefully not only prove to be an effective communication aid, but may provide some language intervention benefits for this population.

In this paper we discuss how generation issues affect the design of a computer-assisted language learning tool designed to teach written English as a second language to deaf users of American Sign Language.We discuss a dual-component linguistic model that attempts to reflect the generation process of the learners.The first model component captures the influence of the first language on the acquisition of the second.The second model component captures the process of second language acquisition itself.The linguistic model helps the system identify errors along with their probable source(s).This information is crucial for effective correction.It is also useful in the response phase of the system to focus tutoring on the errors that are most beneficial to correct.In addition, the linguistic model can be used to tailor the system's realization of its response.In this way, the syntactic constructions generated by the system will provide understandable and positive exemplars of the language features currently being acquired by the leamer.

The goal of this project is to provide a communication tool for people with severe speech and motor impairments (SSMI). The tool will facilitate the formation of syntactically correct sentences in the fewest number of keystrokes. Consider the situation where an individual is using a word-based augmentative communication system—each word is (basically) one keystroke and morphological endings etc. require additional keystrokes. Our prototype system is intended to reduce the burden of the user by allowing him/her to select only the uninflected content words of the desired sentence. The system is responsible for adding proper function words (e.g., articles, prepositions) and necessary morphological endings. In order to accomplish this task, the system attempts to generate a semantic representation of an utterance under circumstances where syntactic (parse tree) information is not available because the input to the system is a compressed telegraphic message rather than a standard English sentence. The representation is used by the system to generate a full English sentence from the compressed input. The focus of the paper is on the knowledge and processing necessary to produce a semantic representation under these telegraphic constraints.

It has been hypothesized that Tree Adjoining Grammar (TAG) is particularly well suited for sentence generation. It is unclear, however, how a sentence generation system based on TAG should choose among the syntactic possibilities made available in the grammar. In this paper we consider the question of what needs to be done to generate with TAGs and explain a generation system that provides the necessary features. This approach is compared with other TAG-based generation systems. Particular attention is given to Mumble-86 which, like our system, makes syntactic choice on sophisticated functional grounds.

As a user interacts with a database or expert system, s/he may reveal a misconception about the objects modeled by the system. This paper discusses the ROMPER system for responding to such misconceptions in a domain independent and context sensitive fashion. ROMPER reasons about possible sources of the misconception. It operates on a model of the user and generates a cooperative response based on this reasoning. The process is made context sensitive by augmenting the user model with a new notion of object perspective which highlights certain aspects of the user model due to previous discourse.

This paper describes a computational method for correcting users' misconceptions concerning the objects modelled by a computer system. The method involves classifying object-related misconceptions according to the knowledge-base feature involved in the incorrect information. For each resulting class sub-types are identified, according to the structure of the knowledge base, which indicate what information may be supporting the misconception and therefore what information to include in the response. Such a characterization, along with a model of what the user knows, enables the system to reason in a domain-independent way about how best to correct the user.

Because people's knowledge is often partial and/or faulty, it is inevitable that misconceptions will be revealed in the course of a conversation. If recognized, the other person may say something to correct the misconception, and the conversation continues. Just as this is the case when people interact with each other, so must it be when users interact with a computer system. For example, in interacting with an expert system, a user may reveal misconceptions about objects modelled by the system. By failing to correct such misconceptions, the system may not only confirm the original misconception, but may cause the user to develop further misconceptions. It must therefore be up to the system to recognize and respond to misconceptions in an effective way. In this paper the space of possible object misconceptions is characterized according to the kind of incorrect information involved. It has been found that this characterization is often useful in determining how the user arrived at the misconception, and therefore the kind of information to include in the response. Using such a characterization, a system will be able to effectively correct object misconceptions in a domain independent way. Factors which affect the amount of information included in a correction (such as discourse and situational context) are also examined.

The knowledge representation is an important factor in natural language generation since it limits the semantic capabilities of the generation system. This paper identifies several information types in a knowledge representation that can be used to generate meaningful responses to questions about database structure. Creating such a knowledge representation, however, is a long and tedious process. A system is presented which uses the contents of the database to form part of this knowledge representation automatically. It employs three types of world knowledge axioms to ensure that the representation formed is meaningful and contains salient information.