Contrastive self-supervised learning based on point-wise comparisons has been widely studied for vision tasks. In the visual cortex of the brain, neuronal responses to distinct stimulus classes are organized into geometric structures known as neural manifolds. Accurate classification of stimuli can be achieved by effectively separating these manifolds, akin to solving a packing problem. We introduce Contrastive Learning As Manifold Packing (CLAMP), a self-supervised framework that recasts representation learning as a manifold packing problem. CLAMP introduces a loss function inspired by the potential energy of short-range repulsive particle systems, such as those encountered in the physics of simple liquids and jammed packings. In this framework, each class consists of sub-manifolds embedding multiple augmented views of a single image. The sizes and positions of the sub-manifolds are dynamically optimized by following the gradient of a packing loss. This approach yields interpretable dynamics in the embedding space that parallel jamming physics, and introduces geometrically meaningful hyperparameters within the loss function. Under the standard linear evaluation protocol, which freezes the backbone and trains only a linear classifier, CLAMP achieves competitive performance with state-of-the-art self-supervised models. Furthermore, our analysis reveals that neural manifolds corresponding to different categories emerge naturally and are effectively separated in the learned representation space, highlighting the potential of CLAMP to bridge insights from physics, neural science, and machine learning.

Brain oscillations might be traveling waves propagating in cortex. Studying their propagation within single cortical areas has mostly been restricted to invasive measurements. Their investigation in healthy humans, however, requires non-invasive recordings, such as MEG or EEG. Identifying traveling waves with these techniques is challenging because source summation, volume conduction, and low signal-to-noise ratios make it difficult to localize cortical activity from sensor responses. The difficulty is compounded by the lack of a known ground truth in traveling wave experiments. Rather than source-localizing cortical responses from sensor activity, we developed a two-part model-based neuroimaging approach: (1) The putative neural sources of a propagating oscillation were modeled within primary visual cortex (V1) via retinotopic mapping from functional MRI recordings (encoding model); and (2) the modeled sources were projected onto MEG and EEG sensors to predict the resulting signal using a biophysical head model. We tested our model by comparing its predictions against the MEG-EEG signal obtained when participants viewed visual stimuli designed to elicit either fovea-to-periphery or periphery-to-fovea traveling waves or standing waves in V1, in which ground truth cortical waves could be reasonably assumed. Correlations on within-sensor phase and amplitude relations between predicted and measured data revealed good model performance. Crucially, the model predicted sensor data more accurately when the input to the model was a traveling wave going in the stimulus direction compared to when the input was a standing wave, or a traveling wave in a different direction. Furthermore, model accuracy peaked at the spatial and temporal frequency parameters of the visual stimulation. Together, our model successfully recovers traveling wave properties in cortex when they are induced by traveling waves in stimuli. This provides a sound basis for using MEG-EEG to study endogenous traveling waves in cortex and test hypotheses related with their role in cognition.

Stability in recurrent neural models poses a significant challenge, particularly in developing biologically plausible neurodynamical models that can be seamlessly trained. Traditional cortical circuit models are notoriously difficult to train due to expansive nonlinearities in the dynamical system, leading to an optimization problem with nonlinear stability constraints that are difficult to impose. Conversely, recurrent neural networks (RNNs) excel in tasks involving sequential data but lack biological plausibility and interpretability. In this work, we address these challenges by linking dynamic divisive normalization (DN) to the stability of "oscillatory recurrent gated neural integrator circuits'' (ORGaNICs), a biologically plausible recurrent cortical circuit model that dynamically achieves DN and that has been shown to simulate a wide range of neurophysiological phenomena. By using the indirect method of Lyapunov, we prove the remarkable property of unconditional local stability for an arbitrary-dimensional ORGaNICs circuit when the recurrent weight matrix is the identity. We thus connect ORGaNICs to a system of coupled damped harmonic oscillators, which enables us to derive the circuit's energy function, providing a normative principle of what the circuit, and individual neurons, aim to accomplish. Further, for a generic recurrent weight matrix, we prove the stability of the 2D model and demonstrate empirically that stability holds in higher dimensions. Finally, we show that ORGaNICs can be trained by backpropagation through time without gradient clipping/scaling, thanks to its intrinsic stability property and adaptive time constants, which address the problems of exploding, vanishing, and oscillating gradients. By evaluating the model's performance on RNN benchmarks, we find that ORGaNICs outperform alternative neurodynamical models on static image classification tasks and perform comparably to LSTMs on sequential tasks.

Statistical whitening transformations play a fundamental role in many computational systems, and may also play an important role in biological sensory systems. Existing neural circuit models of adaptive whitening operate by modifying synaptic interactions; however, such modifications would seem both too slow and insufficiently reversible. Motivated by the extensive neuroscience literature on gain modulation, we propose an alternative model that adaptively whitens its responses by modulating the gains of individual neurons. Starting from a novel whitening objective, we derive an online algorithm that whitens its outputs by adjusting the marginal variances of an overcomplete set of projections. We map the algorithm onto a recurrent neural network with fixed synaptic weights and gain-modulating interneurons. We demonstrate numerically that sign-constraining the gains improves robustness of the network to ill-conditioned inputs, and a generalization of the circuit achieves a form of local whitening in convolutional populations, such as those found throughout the visual or auditory systems.

In interocular suppression, a suprathreshold monocular target can be rendered invisible by a salient competitor stimulus presented in the other eye. Despite decades of research on interocular suppression and related phenomena (e.g., binocular rivalry, flash suppression, continuous flash suppression), the neural processing underlying interocular suppression is still unknown. We developed and tested a computational model of interocular suppression. The model included two processes that contributed to the strength of interocular suppression: divisive normalization and attentional modulation. According to the model, the salient competitor induced a stimulus-driven attentional modulation selective for the location and orientation of the competitor, thereby increasing the gain of neural responses to the competitor and reducing the gain of neural responses to the target. Additional suppression was induced by divisive normalization in the model, similar to other forms of visual masking. To test the model, we conducted psychophysics experiments in which both the size and the eye-of-origin of the competitor were manipulated. For small and medium competitors, behavioral performance was consonant with a change in the response gain of neurons that responded to the target. But large competitors induced a contrast-gain change, even when the competitor was split between the two eyes. The model correctly predicted these results and outperformed an alternative model in which the attentional modulation was eye specific. We conclude that both stimulus-driven attention (selective for location and feature) and divisive normalization contribute to interocular suppression.

Binocular rivalry and cross-orientation suppression are well-studied forms of competition in visual cortex, but models of these two types of competition are in tension with one another. Binocular rivalry occurs during the presentation of dichoptic grating stimuli, where two orthogonal gratings presented separately to the two eyes evoke strong alternations in perceptual dominance. Cross-orientation suppression occurs during the presentation of plaid stimuli, where the responses to a component grating presented to both eyes is weakened by the presence of a superimposed orthogonal grating. Conventional models of rivalry that rely on strong competition between orientation-selective neurons incorrectly predict rivalry between the components of plaids. Lowering the inhibitory weights in such models reduces rivalry for plaids, but also reduces it for dichoptic gratings. Using an exhaustive grid search, we show that this problem cannot be solved simply by adjusting the parameters of the model. Instead, we propose a robust class of models that rely on ocular opponency neurons, previously proposed as a mechanism for efficient stereo coding, to yield rivalry only for dichoptic gratings, not for plaids. This class of models reconciles models of binocular rivalry with the divisive normalization framework that has been used to explain cross-orientation. Our model makes novel predictions that we confirmed with psychophysical tests.

This paper addresses the derivation of likelihood functions and confidence bounds for problems involving over-determined linear systems with noise in all measurements, often referred to as total-least-squares (TLS). It has been shown previously that TLS provides maximum likelihood estimates. But rather than being a function solely of the variables of interest, the associated likelihood functions increase in dimensionality with the number of equations. This has made it difficult to derive suitable confidence bounds, and impractical to use these probability functions with Bayesian belief propagation or Bayesian tracking. This paper derives likelihood functions that are defined only on the parameters of interest. This has two main advantages. First, the likelihood functions are much easier to use within a Bayesian framework; and second it is straightforward to obtain a reliable confidence bound on the estimates. We demonstrate the accuracy of our confidence bound in relation to others that have been proposed. Also, we use our theoretical results to obtain likelihood functions for estimating the direction of 3D camera translation.

We describe a method for embedding information in color images. A model of human color vision is used to ensure that the embedded signal is invisible. Sinusoidal signals are embedded so that they can be detected (decoded) without use of the original image. The sinusoids act as a grid, providing a coordinate frame on the image. We use the grid to automatically scale and align (deskew) images that have been printed and then scanned.

We evaluated six algorithms for computing egomotion from image velocities. We established benchmarks for quantifying bias and sensitivity to noise, and for quantifying the convergence properties of those algorithms that require numerical search. Our simulation results reveal some interesting and surprising results. First, it is often written in the literature that the egomotion problem is difficult because translation (e.g., along the X-axis) and rotation (e.g., about the Y-axis) produce similar image velocities. We found, to the contrary, that the bias and sensitivity of our six algorithms are totally invariant with respect to the axis of rotation. Second, it is also believed by some that fixating helps to make the egomotion problem easier: We found, to the contrary, that fixating does not help when the noise is independent of the image velocities. Fixation does help if the noise is proportional to speed, but this is only for the trivial reason that the speeds are slower under fixation. Third, it is widely believed that increasing the field of view will yield better performance. We found, to the contrary, that this is not necessarily true.

This paper describes a method for synthesizing images that match the texture appearance of a given digitized sample. This synthesis is completely automatic and requires only the "target" texture as input. It allows generation of as much texture as desired so that any object can be covered. The approach is based on a model of human texture perception, and has potential to be a practically useful tool for image processing and graphics applications.

The goal of numerous digital image processing algorithms is to reproduce an image as accurately as possible given some specific restrictions. For example, in digital image halftoning, a gray scale version of an image needs to be approximated by a high spatial resolution binary image. Likewise, lossy image compression seeks to reconstruct an image from a minimally coded description of the original. In these and many other applications, image fidelity is determined by the human observer; hence, the effectiveness of the algorithm is measured by the extent to which reproduction errors are visible. As a result, a model that predicts human perceptual sensitivity to image distortion is beneficial to both the design and evaluation of many such image processing algorithms. This summary briefly describes an extension of our work on perceptual image distortion. Our extended perceptual model accounts for: (1) contrast sensitivity as a function of spatial frequency, mean luminance and spatial extent, (2) luminance masking, and (3) contrast masking.

Article Pyramid-based texture analysis/synthesis Share on Authors: David J. Heeger Stanford University, Department of Psychology, Stanford University, Stanford, CA Stanford University, Department of Psychology, Stanford University, Stanford, CAView Profile , James R. Bergen SRI David Sarnoff Research Center, Princeton, NJ SRI David Sarnoff Research Center, Princeton, NJView Profile Authors Info & Claims SIGGRAPH '95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniquesSeptember 1995 Pages 229–238https://doi.org/10.1145/218380.218446Online:15 September 1995Publication History 520citation4,533DownloadsMetricsTotal Citations520Total Downloads4,533Last 12 Months152Last 6 weeks19 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

In this paper, we present a perceptual distortion measure that predicts image integrity far better than mean-squared error. This perceptual distortion measure is based an a model of human visual processing that fits empirical measurements of the psychophysics of spatial pattern detection. The model of human visual processing proposed involves two major components: a steerable pyramid transform and contrast normalization. We also illustrate the usefulness of the model in predicting perceptual distortion in real images.>

One of the major drawbacks of orthogonal wavelet transforms is their lack of translation invariance: the content of wavelet subbands is unstable under translations of the input signal. Wavelet transforms are also unstable with respect to dilations of the input signal and, in two dimensions, rotations of the input signal. The authors formalize these problems by defining a type of translation invariance called shiftability. In the spatial domain, shiftability corresponds to a lack of aliasing; thus, the conditions under which the property holds are specified by the sampling theorem. Shiftability may also be applied in the context of other domains, particularly orientation and scale. Jointly shiftable transforms that are simultaneously shiftable in more than one domain are explored. Two examples of jointly shiftable transforms are designed and implemented: a 1-D transform that is jointly shiftable in position and scale, and a 2-D transform that is jointly shiftable in position and orientation. The usefulness of these image representations for scale-space analysis, stereo disparity measurement, and image enhancement is demonstrated.>

Gradient methods are widely used in the computation of optical flow. The authors discuss extensions of these methods which compute probability distributions of optical flow. The use of distributions allows representation of the uncertainties inherent in the optical flow computation, facilitating the combination with information from other sources. Distributed optical flow for a synthetic image sequence is computed, and it is demonstrated that the probabilistic model accounts for the errors in the flow estimates. The distributed optical flow for a real image sequence is computed.>

We describe a technique for displaying patterns that appear to move continuously without changing their positions. The method uses a quadrature pair of oriented filters to vary the local phase, giving the sensation of motion. We have used this technique in various computer graphic and scientific visualization applications.

The nonlinear equation that relates the optical-flow field to 3-D motion and depth can be split by an exact algebraic manipulation to form three sets of equations. The first set relates the image velocities to the translational component of the 3-D motion alone. Thus, the depth and the rotational velocity need not be known or estimated prior to solving for the translational velocity. Once the translation has been recovered, the second set of equations can be used to solve for the rotational velocity. Finally, depth can be estimated with the third set of equations, given the recovered transaction and rotation. The algorithm applies to the general case of arbitrary motion with respect to an arbitrary scene. The authors show that the performance of the algorithm compares quite favorably with other proposed approaches.>

As an observer moves and explores the environment, the visual stimulation in his eye is constantly changing. Somehow he is able to perceive the spatial layout of the scene, and to discern his movement through space. Computational vision researchers have been trying to solve this problem for a number of years with only limited success. It is a difficult problem to solve because the relationship between the optical-flow field, the 3D motion parameters, and depth is nonlinear. We have come to understand that this nonlinear equation describing the optical-flow field can be split by an exact algebraic manipulation to yield an equation that relates the image velocities to the translational component of the 3D motion alone. Thus, the depth and the rotational velocity need not be known or estimated prior to solving for the translational velocity. The algorithm applies to the general case of arbitrary motion with respect to an arbitrary scene. It is simple to compute and it is plausible biologically.

This paper formulates the tasks of moving-object detection and motion-based depth recovery as a problem of sensor fusion in the presence of uncertainty. We utilize two sensor systems, one providing information about the local image velocity (spsecifying a point in image-velocity space), and the othter providing information about the camera motion (specifying a line segment in image-velocity space). 1Ne utilize Mahalanobis distance as a threshold rule for determining consistency between the measurements from the two sensor systems. We also suggest a framework for using the resulting segmented flow field to update estimates of the egomotion parameters.