Evaluating deep neural networks (DNNs) as models of human perception has given rich insights into both human visual processing and representational properties of DNNs. We extend this work by analyzing how well DNNs perform compared to humans when constrained by peripheral vision -- which limits human performance on a variety of tasks, but also benefits the visual system significantly. We evaluate this by (1) modifying the Texture Tiling Model (TTM), a well tested model of peripheral vision to be more flexibly used with DNNs, (2) generating a large dataset which we call COCO-Periph that contains images transformed to capture the information available in human peripheral vision, and (3) comparing DNNs to humans at peripheral object detection using a psychophysics experiment. Our results show that common DNNs underperform at object detection compared to humans when simulating peripheral vision with TTM. Training on COCO-Periph begins to reduce the gap between human and DNN performance and leads to small increases in corruption robustness, but DNNs still struggle to capture human-like sensitivity to peripheral clutter. Our work brings us closer to accurately modeling human vision, and paves the way for DNNs to mimic and sometimes benefit from properties of human visual processing.

Computer graphics seeks to deliver compelling images, generated within a computing budget, targeted at a specific display device, and ultimately viewed by an individual user. The foveated nature of human vision offers an opportunity to efficiently allocate computation and compression to appropriate areas of the viewer’s visual field, of particular importance with the rise of high-resolution and wide field-of-view display devices. However, while variations in acuity and contrast sensitivity across the field of view have been well-studied and modeled, a more consequential variation concerns peripheral vision’s degradation in the face of clutter, known as crowding. Understanding of peripheral crowding has greatly advanced in recent years, in terms of both phenomenology and modeling. Accurately leveraging this knowledge is critical for many applications, as peripheral vision covers a majority of pixels in the image. We advance computational models for peripheral vision aimed toward their eventual use in computer graphics. In particular, researchers have recently developed high-performing models of peripheral crowding, known as “pooling” models, which predict a wide range of phenomena but are computationally inefficient. We reformulate the problem as a dataflow computation, which enables faster processing and operating on larger images. Further, we account for the explicit encoding of “end stopped” features in the image, which was missing from previous methods. We evaluate our model in the context of perception of textures in the periphery, including a novel texture dataset and updated textural descriptors. Our improved computational framework may simplify development and testing of more sophisticated, complete models in more robust and realistic settings relevant to computer graphics.

Recent advances in human vision research have pointed toward a theory that unifies many aspects of vision relevant to information visualization. According to this theory, loss of information in peripheral vision determines performance on many visual tasks. This theory subsumes old concepts such as visual saliency, selective attention, and change blindness. It predicts the rich details we have access to at a glance. Furthermore, it provides insight into tasks not commonly studied in human vision, such as ability to comprehend connections in a network diagram, or to compare information in one part of a display with that in another.

In crowding, perception of an object deteriorates in the presence of nearby elements. Although crowding is a ubiquitous phenomenon, since elements are rarely seen in isolation, to date there exists no consensus on how to model it. Previous experiments showed that the global configuration of the entire stimulus must be taken into account. These findings rule out simple pooling or substitution models and favor models sensitive to global spatial aspects. In order to investigate how to incorporate global aspects into models, we tested a large number of models with a database of forty stimuli tailored for the global aspects of crowding. Our results show that incorporating grouping like components strongly improves model performance.

Navigating in a previously unknown environment and recognizing naturally occurring text in a scene are two important autonomous capabilities that are typically treated as distinct. However, these two tasks are potentially complementary, (i) scene and pose priors can benefit text spotting, and (ii) the ability to identify and associate text features can benefit navigation accuracy through loop closures. Previous approaches to autonomous text spotting typically require significant training data and are too slow for real-time implementation. In this work, we propose a novel high-level feature descriptor, the “junction”, which is particularly well-suited to text representation and is also fast to compute. We show that we are able to improve SLAM through text spotting on datasets collected with a Google Tango, illustrating how location priors enable improved loop closure with text features.

Understanding and exploiting the abilities of the human visual system is an important part of the design of usable user interfaces and information visualizations. Designers traditionally learn qualitative rules of thumb for how to enable quick, easy, and veridical perception of their design. More recently, work in human and computer vision has produced more quantitative models of human perception, which take as input arbitrary, complex images of a design. In this article, we ask whether models of perception aid the design process, using our tool DesignEye as a working example of a perceptual tool incorporating such models. Through a series of interactions with designers and design teams, we find that the models can help, but in somewhat unexpected ways. DesignEye was capable of facilitating A/B comparisons between designs, and judgments about the quality of a design. However, overall “goodness” values were not very useful, showed signs of interfering with a natural process of trading off perceptual vs. other design issues, and would likely interfere with acceptance of a perceptual tool by professional designers. Perhaps most surprisingly, DesignEye, by providing in essence a simple visualization of the design, seemed to facilitate communication about not only perceptual aspects of design, but also about design goals and how to achieve those goals. We discuss resulting design principles for making perceptual tools useful in general.

We are interested in identifying the material category, e.g. glass, metal, fabric, plastic or wood, from a single image of a surface. Unlike other visual recognition tasks in computer vision, it is difficult to find good, reliable features that can tell material categories apart. Our strategy is to use a rich set of low and mid-level features that capture various aspects of material appearance. We propose an augmented Latent Dirichlet Allocation (aLDA) model to combine these features under a Bayesian generative framework and learn an optimal combination of features. Experimental results show that our system performs material recognition reasonably well on a challenging material database, outperforming state-of-the-art material/texture recognition systems.

Understanding and exploiting the abilities of the human visual system is an important part of the design of usable user interfaces and information visualizations. Good design enables quick, easy and veridical perception of key components of that design. An important facet of human vision is its ability to seemingly effortlessly perform "perceptual organization; it transforms individual feature estimates into perception of coherent regions, structures, and objects. We perceive regions grouped by proximity and feature similarity, grouping of curves by good continuation, and grouping of regions of coherent texture. In this paper, we discuss a simple model for a broad range of perceptual grouping phenomena. It takes as input an arbitrary image, and returns a structure describing the predicted visual organization of the image. We demonstrate that this model can capture aspects of traditional design rules, and predicts visual percepts in classic perceptual grouping displays.

Management of clutter is an important factor in the design of user interfaces and information visualizations, allowing improved usability and aesthetics. However, clutter is not a well defined concept. In this paper, we present the Feature Congestion measure of display clutter. This measure is based upon extensive modeling of the saliency of elements of a display, and upon a new operational definition of clutter. The current implementation is based upon two features: color and luminance contrast. We have tested this measure on maps that observers ranked by perceived clutter. Results show good agreement between the observers' rankings and our measure of clutter. Furthermore, our measure can be used to make design suggestions in an automated UI critiquing tool.

Although existing GUIs have a sense of space, they provide no sense of place. Numerous studies report that users misplace files and have trouble wayfinding in virtual worlds despite the fact that people have remarkable visual and spatial abilities. This issue is considered in the human-computer interface field and has been addressed with alternate display/navigation schemes. Our paper presents a fundamentally graphics based approach to this 'lost in hyperspace' problem. Specifically, we propose that spatial display of files is not sufficient to engage our visual skills; scenery (distinctive visual appearance) is needed as well. While scenery (in the form of custom icon assignments) is already possible in current operating systems, few if any users take the time to manually assign icons to all their files. As such, our proposal is to generate visually distinctive icons ("VisualIDs") automatically , while allowing the user to replace the icon if desired. The paper discusses psychological and conceptual issues relating to icons, visual memory, and the necessary relation of scenery to data. A particular icon generation algorithm is described; subjects using these icons in simulated file search and recall tasks show significantly improved performance with little effort. Although the incorporation of scenery in a graphical user interface will introduce many new (and interesting) design problems that cannot be addressed in this paper, we show that automatically created scenery is both beneficial and feasible.

As users pan and zoom, display content can disappear into off-screen space, particularly on small-screen devices. The clipping of locations, such as relevant places on a map, can make spatial cognition tasks harder. Halo is a visualization technique that supports spatial cognition by showing users the location of off-screen objects. Halo accomplishes this by surrounding off-screen objects with rings that are just large enough to reach into the border region of the display window. From the portion of the ring that is visible on-screen, users can infer the off-screen location of the object at the center of the ring. We report the results of a user study comparing Halo with an arrow-based visualization technique with respect to four types of map-based route planning tasks. When using the Halo interface, users completed tasks 16-33% faster, while there were no significant differences in error rate for three out of four tasks in our study.

We present an overview+detail document interface that draws on perceptual principles to help users work with documents. Central to our approach is the use of improved document overviews. Our approach also includes novel highlighting in the full representation of documents, as well as techniques to help users smoothly transition from the overview to the full representation of the document. We present a specific implementation of our design for Web browsing. We also present a qualitative user study that indicates that our perceptual design principles are effective and that users prefer our interface to traditional "find" and highlighting techniques. Our user study additionally reveals interesting tasks and strategies supported in our framework that have implications for overview+detail document interfaces in general

Abstract We introduce a technique for creating novel, enhanced thumbnails of Web pages. These thumbnails combine the advantages of plain thumbnails and text summaries to provide consistent performance on a variety of tasks. We conducted a study in which participants used three different types of summaries (enhanced thumbnails, plain thumbnails, and text summaries) to search Web pages to find several different types of information. Participants took an average of 67, 86, and 95 seconds to find the answer with enhanced thumbnails, plain thumbnails, and text summaries, respectively. As expected, there was a strong effect of question category. For some questions, text summaries outperformed plain thumbnails, while for other questions, plain thumbnails outperformed text summaries. Enhanced thumbnails (which combine the features of text summaries and plain thumbnails) had more consistent performance than either text summaries or plain thumbnails, having for all categories the best performance or performance that was statistically indistinguishable from the best.

We introduce a technique for creating novel, textually-enhanced thumbnails of Web pages. These thumbnails combine the advantages of image thumbnails and text summaries to provide consistent performance on a variety of tasks. We conducted a study in which participants used three different types of summaries (enhanced thumbnails, plain thumbnails, and text summaries) to search Web pages to find several different types of information. Participants took an average of 67, 86, and 95 seconds to find the answer with enhanced thumbnails, plain thumbnails, and text summaries, respectively. We found a strong effect of question category. For some questions, text outperformed plain thumbnails, while for other questions, plain thumbnails outperformed text. Enhanced thumbnails (which combine the features of text summaries and plain thumbnails) were more consistent than either text summaries or plain thumbnails, having for all categories the best performance or performance that was statistically indistinguishable from the best.

Motion of an observer relative to objects in a scene provides information about the structure of the scene. Changing patterns of shading due to motion relative to the light source provide information about surface structure, albedos, and light sources. One can stratify this photometric information into affine, unitary, and metric structure, much like the stratification of structure from motion. For Lambertian surfaces, if either motion or photometry give us more than affine structure, the two cues can be combined to yield full metric information. Edge constraints plus unitary photometry also give us full metric photometry. Affine structure alone contains much of the quantitative structure information, allowing us to judge such things as the ordinal relationships between the albedos.

An extension to the JPEG standard (ISO/IEC DIS 10918-3) allows spatial adaptive coding of still images. As with baseline JPEG coding, one quantization matrix applies to an entire image channel, but in addition the user may specify a multiplier for each 8/spl times/8 block, which multiplies the quantization matrix, yielding the new matrix for that block. MPEG 1 and 2 use much the same scheme, except there the multiplier changes only on macroblock boundaries. We propose a method for perceptual optimization of the set of multipliers. We compute the perceptual error for each block based upon the DCT quantization error adjusted according to the contrast sensitivity, light adaptation, and contrast masking, and pick the set of multipliers which yield maximally flat perceptual error over the blocks of the image. We investigate the bit rate savings due to this adaptive coding scheme and the relative importance of the different sorts of masking on adaptive coding.

Shape from texture is best analyzed in a two-stage framework: computing the texture gradient from the image, and interpreting the texture gradient to infer the distance and shape of the scene surface in 3-D. The texture distortion is modeled at a point in any particular direction on the image plane as an affine transformation. The parameters of the affine transform are related to the viewer. A technique is developed for estimating affine transforms between nearby image patches. It is based on solving a system of linear constraints derived from a differential analysis. Given the estimated affine transforms in a number of directions, it is possible to recover local shape and orientation of the surface. It is not necessary to explicitly identify texels or make restrictive assumptions about the nature of the image texture. Experimental results are presented on images of planar and curved surfaces under perspective projection.>