In this article, we present a novel approach to quantitative evaluation of a model for parsing web pages as visual images, intended to provide improvements for users with assistive needs (cognitive or visual deficits, enabling decluttering or zooming and supporting more effective screen reader output). This segmentation-classification pipeline is tested in stages: We first discuss the validation of the segmentation algorithm, showing that our approach produces automated segmentations that are very similar to those produced by real users when making use of a drawing interface to designate edges and regions. We also examine the properties of these ground truth segmentations produced under different conditions. We then describe our Hidden Markov tree approach for classification and present results which serve provide important validation for this model. The analysis is set against effective choices for dataset and pruning options, measured with respect to manual ground truth labelling of regions. In all, we offer a detailed quantitative validation (focused on complex news pages) of a fully pipelined approach for interpreting web pages as visual images, an approach which enables important advances for users with assistive needs.

In this paper we present an overview of our proposed algorithms for classifying regions of web pages based on content and visual properties. We show how hidden Markov trees may be effective for the classification and how this may end up offering improved experiences to users who are trying to view webpages.

This paper presents a recognizer for identifying references to user interface components in online documentation. The recognizer first extracts phrases matching a list of known components, then employs a classifier to reject coincidental matches. We describe why this seemingly straightforward problem is challenging, then show how informal conventions in documentation writing can be leveraged to perform classification. Using the features identified in this paper, our approach achieves an average F1 score of 0.81, and can correctly distinguish between actual command references and coincidental matches in 93.7% of test cases.

People routinely rely on Internet search engines to support their use of interactive systems: they issue queries to learn how to accomplish tasks, troubleshoot problems, and otherwise educate themselves on products. Given this common behavior, we argue that search query logs can usefully augment traditional usability methods by revealing the primary tasks and needs of a product's user population. We term this use of search query logs CUTS - characterizing usability through search. In this paper, we introduce CUTS and describe an automated process for harvesting, ordering, labeling, filtering, and grouping search queries related to a given product. Importantly, this data set can be assembled in minutes, is timely, has a high degree of ecological validity, and is arguably less prone to self-selection bias than data gathered via traditional usability methods. We demonstrate the utility of this approach by applying it to a number of popular software and hardware systems.

Recognizing hand drawn mathematical matrices on tablet computers has proven to be a particularly challenging task. While individual expression recognition can be simplified by assuming the entire content is a single semantic construct, a single math expression, a matrix is composed of multiple expressions arranged in rows and columns. These expressions must first be segmented into matrix elements, and then each individual matrix element expression must be recognized. In this work, we show how a simple algorithm on in-air (i.e. non-inking) strokes can be used to analyze the drawing order of a matrix. Once the drawing order is recognized, we show how outlier analysis on in-air packets gives rapid, reliable segmentation of matrix elements.

This paper introduces query-feature graphs, or QF-graphs. QF-graphs encode associations between high-level descriptions of user goals (articulated as natural language search queries) and the specific features of an interactive system relevant to achieving those goals. For example, a QF-graph for the GIMP graphics manipulation software links the query "GIMP black and white" to the commands "desaturate" and "grayscale." We demonstrate how QF-graphs can be constructed using search query logs, search engine results, web page content, and localization data from interactive systems. An analysis of QF-graphs shows that the associations produced by our approach exhibit levels of accuracy that make them eminently usable in a range of real-world applications. Finally, we present three hypothetical user interface mechanisms that illustrate the potential of QF-graphs: search-driven interaction, dynamic tooltips, and app-to-app analogy search.

Many registration scenarios involve aligning more than just two images. These image sets-called ensembles-are conventionally registered by choosing one image as a template, and every other image is registered to it. This pairwise approach is problematic because results depend on which image is chosen as the template. The issue is particularly acute for multisensor ensembles because different sensors create images with different features. Also, pairwise methods use only a fraction of the available data at a time. In this paper, we propose a maximum-likelihood clustering method that registers all the images in a multisensor ensemble simultaneously. Experiments involving rigid-body and affine transformations show that the clustering method is more robust and accurate than competing pairwise registration methods. Moreover, the clustering results can be used to form a rudimentary segmentation of the image ensemble.

Many applications in computer vision use Principal Components Analysis (PCA), for example, in camera calibration, stereo, localization and motion estimation. We present a new and fast PCA-based method to analyze optical snow. Optical snow is a complex form of visual motion that occurs when an observer moves through a highly cluttered 3D scene. For this category of motion field, no spatial or depth coherence can be assumed. Previous methods for measuring optical snow have used a wedge filter in a spatiotemporal frequency domain. The PCA method is also based on the spatiotemporal frequency domain analysis, but examines a different geometry property of the spectrum. We compare the results of the PCA method to the previous methods using both real and synthetic sequences. 1

Multiple differently exposed pictures of the same subject matter arise naturally whenever a video camera having automatic exposure captures multiple frames of video with the same subject matter appearing in regions of overlap between at least some of the successive video frames. Almost all cameras have some kind of automatic exposure feature. Generally automatic exposure is center weighted, so that when a light object falls in the center of the frame the exposure is automatically decreased, whereas the exposure is automatically increased when the camera swings around to point at a darker object. In this paper, it is assumed that the spatial (e.g. projective) coordinate transformation between successive frames of the sequence is known (or equivalently that it is the identity), and the contribution of the paper is an efficient way to estimate the tonal relationship between successive frames of the sequence. In particular methods are proposed to simultaneously estimate the unknown camera response function, as well as the set of unknown relative exposure changes among images, up to a single unknown scalar constant. The method comprises a succession of guesses each of which is a refinement of the previous. The first guess is often sufficient, so that no initial solution needs to be provided by the user. Each subsequent guess is a least squares solution so that no computationally expensive optimization is required. Since the method makes use of all the data, it is extremely immune to noise. The method is tested against state-of-the art laboratory measurement instruments to confirm the accuracy of the results.

Two basic problems in image interpretation are: a) determining which interpretations are the most plausible amongst many possibilities; and b) controlling the search for plausible interpretations. We address these issues using a Bayesian approach, with the plausibility ordering and search pruning based on the posterior probabilities of interpretations. However, due to the need for detailed quantitative prior probabilities and the need to evaluate complex integrals over various conditional distributions, a full Bayesian approach is currently impractical except in tightly constrained domains. To circumvent these difficulties we introduce the notion of qualitative probabilistic analysis. In particular, given spatial and contrast resolution parameters, we consider only the asymptotic order of the posterior probability for any interpretation as these resolutions are made finer. We introduce this approach for a simple card-world domain, and present computational results for blocks-world images.

Understanding observations of interacting objects requires one to reason about qualitative scene dynamics. For example, on observing a hand lifting a can, we may infer that an 'active' hand is applying an upwards force (by grasping) to lift a 'passive' can. Previously we presented a system that infers qualitative scene dynamics from the instantaneous motion of objects. However; since that analysis only considered single frames in isolation, there were often multiple interpretations for each frame. In this work we show how the dynamic information inferred at each frame can be integrated over time to reduce ambiguity. Our approach to integrating information is to extend our representation to describe objects by a set of properties or capabilities that are assumed to persist over time. Given this extended representation we find interpretations that require the smallest set(s) of properties over the whole image sequence.

The results of an experimental study aimed at the classification of radar clutter encountered on ground-based coherent scanning radar systems used for air traffic control are presented. The clutter signals of interest are primarily those due to birds and to clouds and weather systems. A historical perspective on the radar clutter classification problem is given, and related issues are discussed. The important features of radar as a sensor in an air traffic control environment are described, and physical phenomena in radar clutter and targets, which provide the physical basis for the discrimination between the different radar clutter classes, are discussed. The feature of selection/extraction procedure, which is based on the multisegment Burg algorithm, is described. The experimental evaluation of a parametric Bayes classifier and a neural network classifier is reported.>

The authors describe a high-speed systolic-based computer system designed to process real-life radar data in near real time. At the heart of the system is a ten-cell Warp (systolic) machine. A digital recorder is used to input data to the system, and a color monitor displays the output of the processing algorithms. The system permits the processing of large volumes of radar data in real time, displays the results of advanced signal-processing algorithms, and has the effect of bringing a radar environment inside the laboratory for experimental studies or, equivalently, taking a sophisticated computer outside into an operational radar environment. The authors briefly describe research efforts built around this facility and highlight the elements of the powerful signal processing neural network software that are basic to their execution.>