Judging the similarity of visualizations is crucial to various applications, such as visualization-based search and visualization recommendation systems.Recent studies show deep-feature-based similarity metrics correlate well with perceptual judgments of image similarity and serve as effective loss functions for tasks like image super-resolution and style transfer.We explore the application of such metrics to judgments of visualization similarity.We extend a similarity metric using five ML architectures and three pre-trained weight sets.We replicate results from previous crowdsourced studies on scatterplot and visual channel similarity perception.Notably, our metric using pre-trained ImageNet weights outperformed gradient-descent tuned MS-SSIM, a multi-scale similarity metric based on luminance, contrast, and structure.Our work contributes to understanding how deep-feature-based metrics can enhance similarity assessments in visualization, potentially improving visual analysis tools and techniques.Supplementary materials are available at https://osf.io/dj2ms/.

We introduce Focal Split, a handheld, snapshot depth camera with fully onboard power and computing based on depth-from-differential-defocus (DfDD). Focal Split is passive, avoiding power consumption of light sources. Its achromatic optical system simultaneously forms two differentially defocused images of the scene, which can be independently captured using two photosensors in a snapshot. The data processing is based on the DfDD theory, which efficiently computes a depth and a confidence value for each pixel with only 500 floating point operations (FLOPs) per pixel from the camera measurements. We demonstrate a Focal Split prototype, which comprises a handheld custom camera system connected to a Raspberry Pi 5 for real-time data processing. The system consumes 4.9 W and is powered on a 5 V, 10,000 mAh battery. The prototype can measure objects with distances from 0.4 m to 1.2 m, outputting 480×360 sparse depth maps at 2.1 frames per second (FPS) using unoptimized Python scripts. Focal Split is DIY friendly. A comprehensive guide to building your own Focal Split depth camera, code, and additional data can be found at https://focal-split.qiguo.org.

Abstract We propose depth from coupled optical differentiation, a low-computation passive-lighting 3D sensing mechanism. It is based on our discovery that per-pixel object distance can be rigorously determined by a coupled pair of optical derivatives of a defocused image using a simple, closed-form relationship. Unlike previous depth-from-defocus (DfD) methods that leverage higher-order spatial derivatives of the image to estimate scene depths, the proposed mechanism’s use of only first-order optical derivatives makes it significantly more robust to noise. Furthermore, unlike many previous DfD algorithms with requirements on aperture code, this relationship is proved to be universal to a broad range of aperture codes. We build the first 3D sensor based on depth from coupled optical differentiation. Its optical assembly includes a deformable lens and a motorized iris, which enables dynamic adjustments to the optical power and aperture radius. The sensor captures two pairs of images: one pair with a differential change of optical power and the other with a differential change of aperture scale. From the four images, a depth and confidence map can be generated with only 36 floating point operations per output pixel (FLOPOP), more than ten times lower than the previous lowest passive-lighting depth sensing solution to our knowledge. Additionally, the depth map generated by the proposed sensor demonstrates more than twice the working range of previous DfD methods while using significantly lower computation.

Neurons throughout the brain modulate their firing rate lawfully in response to sensory input. Theories of neural computation posit that these modulations reflect the outcome of a constrained optimization in which neurons aim to robustly and efficiently represent sensory information. Our understanding of how this optimization varies across different areas in the brain, however, is still in its infancy. Here, we show that neural sensory responses transform along the dorsal stream of the visual system in a manner consistent with a transition from optimizing for information preservation towards optimizing for perceptual discrimination. Focusing on the representation of binocular disparities-the slight differences in the retinal images of the two eyes-we re-analyze measurements characterizing neuronal tuning curves in brain areas V1, V2, and MT (middle temporal) in the macaque monkey. We compare these to measurements of the statistics of binocular disparity typically encountered during natural behaviors using a Fisher Information framework. The differences in tuning curve characteristics across areas are consistent with a shift in optimization goals: V1 and V2 population-level responses are more consistent with maximizing the information encoded about naturally occurring binocular disparities, while MT responses shift towards maximizing the ability to support disparity discrimination. We find that a change towards tuning curves preferring larger disparities is a key driver of this shift. These results provide new insight into previously-identified differences between disparity-selective areas of cortex and suggest these differences play an important role in supporting visually-guided behavior. Our findings emphasize the need to consider not just information preservation and neural resources, but also relevance to behavior, when assessing the optimality of neural codes.

Robust and non-destructive material classification is a challenging but crucial first-step in numerous vision applications. We propose a physics-guided material classification framework that relies on thermal properties of the object. Our key observation is that the rate of heating and cooling of an object depends on the unique intrinsic properties of the material, namely the emissivity and diffusivity. We leverage this observation by gently heating the objects in the scene with a low-power laser for a fixed duration and then turning it off, while a thermal camera captures measurements during the heating and cooling process. We then take this spatial and temporal “thermal spread function” (TSF) to solve an inverse heat equation using the finite-differences approach, resulting in a spatially varying estimate of diffusivity and emissivity. These tuples are then used to train a classifier that produces a fine-grained material label at each spatial pixel. Our approach is extremely simple requiring only a small light source (low power laser) and a thermal camera, and produces robust classification results with 86% accuracy over 16 classes11Code: https://github.com/aniketdashpute/TSF.

The Transport of Intensity Equation (TIE) in microscopy and the Depth from Differential Defocus (DfDD) method in photography both describe the effect of a small change in defocus on image intensity. They are based on different assumptions and may appear to contradict each other. Using the Wigner Distribution Function, we show that DfDD can be interpreted as a special case of the TIE, well-suited to applications where the generalized phase measurements recovered by the TIE are connected to depth rather than phase, such as photography and fluorescence microscopy. The level of spatial coherence is identified as the driving factor in the trade-off between the usefulness of each technique. Specifically, the generalized phase corresponds to the sample's phase under high-coherence illumination and reveals scene depth in low-coherence settings. When coherence varies spatially, as in multi-modal phase and fluorescence microscopy, we show that complementary information is available in different regions of the image.

Current eye-tracking techniques rely primarily on video-based tracking of components of the anterior surfaces of the eye. However, these trackers have several limitations. Their limited resolution precludes study of small fixational eye motion. Furthermore, many of these trackers rely on calibration procedures that do not offer a way to validate their eye motion traces. By comparison, retinal-image-based trackers can track the motion of the retinal image directly, at frequencies greater than 1kHz and with subarcminute accuracy. The retinal image provides a way to validate the eye position at any point in time, offering an unambiguous record of eye motion as a reference for the eye trace. The benefits of using scanning retinal imaging systems as eye trackers, however, comes at the price of different problems that are not present in video-based systems, and need to be solved to obtain robust eye traces. The current abstract provides an overview of retinal-image-based eye tracking methods, provides preliminary eye-tracking results from a tracking scanning-laser ophthalmoscope (TSLO), and proposes a new binocular line-scanning eye-tracking system.

The focal track sensor is a monocular and computationally efficient depth sensor that is based on defocus controlled by a liquid membrane lens. It synchronizes small lens oscillations with a photosensor to produce real-time depth maps by means of differential defocus, and it couples these oscillations with bigger lens deformations that adapt the defocus working range to track objects over large axial distances. To create the focal track sensor, we derive a texture-invariant family of equations that relate image derivatives to scene depth when a lens changes its focal length differentially. Based on these equations, we design a feed-forward sequence of computations that: robustly incorporates image derivatives at multiple scales; produces confidence maps along with depth; and can be trained endto- end to mitigate against noise, aberrations, and other non-idealities. Our prototype with 1-inch optics produces depth and confidence maps at 100 frames per second over an axial range of more than 75cm.

Robots that engage in social behaviors benefit greatly from possessing tools that allow them to manipulate the course of an interaction. Using a non-anthropomorphic social robot and a simple counting game, we examine the effects that empathy-generating robot dialogue has on participant performance across three conditions. In the self-directed condition, the robot petitions the participant to reduce his or her performance so that the robot can avoid punishment. In the externally-directed condition, the robot petitions on behalf of its programmer so that its programmer can avoid punishment. The control condition does not involve any petitions for empathy. We find that externally-directed petitions from the robot show a higher likelihood of motivating the participant to sacrifice his or her own performance to help, at the expense of incurring negative social effects. We also find that experiencing these emotional dialogue events can have complex and difficult to predict effects, driving some participants to antipathy, leaving some unaffected, and manipulating others into feeling empathy towards the robot.