Neuroimaging software methods are complex, making it a near certainty that some implementations will contain errors. Modern computational techniques (i.e., public code and data repositories, continuous integration, containerization) enable the reproducibility of the analyses and reduce coding errors, but they do not guarantee the scientific validity of the results. It is difficult, nay impossible, for researchers to check the accuracy of software by reading the source code; ground truth test datasets are needed. Computational reproducibility means providing software so that for the same input anyone obtains the same result, right or wrong. Computational validity means obtaining the right result for the ground-truth test data. We describe a framework for validating and sharing software implementations, and we illustrate its usage with an example application: population receptive field (pRF) methods for functional MRI data. The framework is composed of three main components implemented with containerization methods to guarantee computational reproducibility. In our example pRF application, those components are: (1) synthesis of fMRI time series from ground-truth pRF parameters, (2) implementation of four public pRF analysis tools and standardization of inputs and outputs, and (3) report creation to compare the results with the ground truth parameters. The framework was useful in identifying realistic conditions that lead to imperfect parameter recovery in all four pRF implementations, that would remain undetected using classic validation methods. We provide means to mitigate these problems in future experiments. A computational validation framework supports scientific rigor and creativity, as opposed to the oft-repeated suggestion that investigators rely upon a few agreed upon packages. We hope that the framework will be helpful to validate other critical neuroimaging algorithms, as having a validation framework helps (1) developers to build new software, (2) research scientists to verify the software's accuracy, and (3) reviewers to evaluate the methods used in publications and grants.

There is widespread interest in estimating the fluorescence properties of natural materials in an image. However, the separation between reflected and fluoresced components is difficult, because it is impossible to distinguish reflected and fluoresced photons without controlling the illuminant spectrum. We show how to jointly estimate the reflectance and fluorescence from a single set of images acquired under multiple illuminants. We present a framework based on a linear approximation to the physical equations describing image formation in terms of surface spectral reflectance and fluorescence due to multiple fluorophores. We relax the non-convex, inverse estimation problem in order to jointly estimate the reflectance and fluorescence properties in a single optimization step. We provide a software implementation of the solver for our method and prior methods. We evaluate the accuracy and reliability of the method using both simulations and experimental data. To evaluate the methods experimentally we built a custom imaging system using a monochrome camera, a filter wheel with bandpass transmissive filters and a small number of light emitting diodes. We compared the methods based upon our framework with the ground truth as well as with prior methods.

There are many scientific, medical and industrial imaging applications where users have full control of the scene illumination and color reproduction is not the primary objective For example, it is possible to co-design sensors and spectral illumination in order to classify and detect changes in biological tissues, organic and inorganic materials, and object surface properties. In this paper, we propose two different approaches to illuminant spectrum selection for surface classification. In the supervised framework we formulate a biconvex optimization problem where we alternate between optimizing support vector classifier weights and optimal illuminants. We also describe a sparse Principal Component Analysis (PCA) dimensionality reduction approach that can be used with unlabeled data. We efficiently solve the non-convex PCA problem using a convex relaxation and Alternating Direction Method of Multipliers (ADMM). We compare the classification accuracy of a monochrome imaging sensor with optimized illuminants to the classification accuracy of conventional RGB cameras with natural broadband illumination.

Motion parallax is an important cue for depth perception. Rendering it accurately can lead to a more natural and immersive virtual reality (VR) experience. We introduce Stacked Omnistereo, a novel data representation that can render immersive video with six degrees of freedom (DoF). We compare the proposed representation against other depth-based and image-based motion parallax techniques using natural as well as synthetic scenes. We show that the proposed representation can synthesize plausible, view-dependent specular highlights, is compact compared to light fields, and outperforms state-of-the-art VR representations by up to 3 dB when evaluated with 6 DoF head motion.

Many creative ideas are being proposed for image sensor designs, and these may be useful in applications ranging from consumer photography to computer vision. To understand and evaluate each new design, we must create a corresponding image processing pipeline that transforms the sensor data into a form, that is appropriate for the application. The need to design and optimize these pipelines is time-consuming and costly. We explain a method that combines machine learning and image systems simulation that automates the pipeline design. The approach is based on a new way of thinking of the image processing pipeline as a large collection of local linear filters. We illustrate how the method has been used to design pipelines for novel sensor architectures in consumer photography applications.

Tractography uses diffusion MRI to estimate the trajectory and cortical projection zones of white matter fascicles in the living human brain. There are many different tractography algorithms and each requires the user to set several parameters, such as curvature threshold. Choosing a single algorithm with specific parameters poses two challenges. First, different algorithms and parameter values produce different results. Second, the optimal choice of algorithm and parameter value may differ between different white matter regions or different fascicles, subjects, and acquisition parameters. We propose using ensemble methods to reduce algorithm and parameter dependencies. To do so we separate the processes of fascicle generation and evaluation. Specifically, we analyze the value of creating optimized connectomes by systematically combining candidate streamlines from an ensemble of algorithms (deterministic and probabilistic) and systematically varying parameters (curvature and stopping criterion). The ensemble approach leads to optimized connectomes that provide better cross-validated prediction error of the diffusion MRI data than optimized connectomes generated using a single-algorithm or parameter set. Furthermore, the ensemble approach produces connectomes that contain both short- and long-range fascicles, whereas single-parameter connectomes are biased towards one or the other. In summary, a systematic ensemble tractography approach can produce connectomes that are superior to standard single parameter estimates both for predicting the diffusion measurements and estimating white matter fascicles.

Many multispectral imaging systems are computational in nature and require processing of raw data in order to obtain radiance spectra. In this paper, we derive a fast and scalable spectral estimation algorithm based on the Alternating Direction Method of Multipliers (ADMM). Using this approach we solve for the unknown surface spectral reflectance simultaneously for all pixels in the image. This global formulation allows us to incorporate spatial as well as spectral regularizers, such as total variation penalty or non-negativity. We show that the estimates derived with our solver are more accurate and more robust in the presence of noise.

Visual neuroscientists have discovered fundamental properties of neural representation through careful analysis of responses to controlled stimuli. Typically, different properties are studied and modeled separately. To integrate our knowledge, it is necessary to build general models that begin with an input image and predict responses to a wide range of stimuli. In this study, we develop a model that accepts an arbitrary band-pass grayscale image as input and predicts blood oxygenation level dependent (BOLD) responses in early visual cortex as output. The model has a cascade architecture, consisting of two stages of linear and nonlinear operations. The first stage involves well-established computations-local oriented filters and divisive normalization-whereas the second stage involves novel computations-compressive spatial summation (a form of normalization) and a variance-like nonlinearity that generates selectivity for second-order contrast. The parameters of the model, which are estimated from BOLD data, vary systematically across visual field maps: compared to primary visual cortex, extrastriate maps generally have larger receptive field size, stronger levels of normalization, and increased selectivity for second-order contrast. Our results provide insight into how stimuli are encoded and transformed in successive stages of visual processing.

Multispectral scene information is useful for radiometric graphics, material identification and imaging systems simulation. The multispectral scene can be described as a datacube, which is a 3D representation of energy at multiple wavelength samples at each scene spatial location. Typically, multispectral scene data are acquired using costly methods that either employ tunable filters or light sources to capture multiple narrow-bands of the spectrum at each spatial point. In this paper, we present new computational methods that estimate the datacube from measurements with a conventional digital camera. Existing methods reconstruct spectra at single locations independently of their neighbors. In contrast, we present a method that jointly recovers the spatio-spectral datacube by exploiting the data sparsity in a transform representation.

The appearance of rendered text is a compromise between the designer's intent and the display capabilities. The ClearType rendering method is designed to enhance rendered text by exploiting the subpixel resolution available on color displays. ClearType represents the high-resolution font outline at the full subpixel resolution of the display and then filters the image to enhance contrast and reduce color artifacts. The filter choice influences text appearance, and people have clear preferences between the renderings with different filters. In this paper, we predict these preferences using S-CIELAB, a spatial extension to the perceptual color metric CIELAB. We calculate the S-CIELAB difference between designed and rendered fonts for various filters. We compare the size of these differences with preference data obtained from individual subjects.

Diffusion Tensor Imaging (DTI) is a magnetic resonance imaging method that can be used to measure local information about the structure of white matter within the human brain. Combining DTI data with the computational methods of MR tractography, neuroscientists can estimate the locations and sizes of nerve bundles (white matter pathways) that course through the human brain. Neuroscientists have used visualization techniques to better understand tractography data, but they often struggle with the abundance and complexity of the pathways. In this paper, we describe a novel set of interaction techniques that make it easier to explore and interpret such pathways. Specifically, our application allows neuroscientists to place and interactively manipulate box or ellipsoid-shaped regions to selectively display pathways that pass through specific anatomical areas. These regions can be used in coordination with a simple and flexible query language which allows for arbitrary combinations of these queries using Boolean logic operators. A representation of the cortical surface is provided for specifying queries of pathways that may be relevant to gray matter structures and for displaying activation information obtained from functional magnetic resonance imaging. By precomputing the pathways and their statistical properties, we obtain the speed necessary for interactive question-and-answer sessions with brain researchers. We survey some questions that researchers have been asking about tractography data and show how our system can be used to answer these questions efficiently.

Diffusion tensor imaging (DTI) is a magnetic resonance imaging method that can be used to measure local information about the structure of white matter within the human brain. Combining DTI data with the computational methods of MR tractography, neuroscientists can estimate the locations and sizes of nerve bundles (white matter pathways) that course through the human brain. Neuroscientists have used visualization techniques to better understand tractography data, but they often struggle with the abundance and complexity of the pathways. We describe a novel set of interaction techniques that make it easier to explore and interpret such pathways. Specifically, our application allows neuroscientists to place and interactively manipulate box-shaped regions (or volumes of interest) to selectively display pathways that pass through specific anatomical areas. A simple and flexible query language allows for arbitrary combinations of these queries using Boolean logic operators. Queries can be further restricted by numerical path properties such as length, mean fractional anisotropy, and mean curvature. By precomputing the pathways and their statistical properties, we obtain the speed necessary for interactive question-and-answer sessions with brain researchers. We survey some questions that researchers have been asking about tractography data and show how our system can be used to answer these questions efficiently.

This paper describes practical algorithms and experimental results concerning illuminant classification. Specifically, we review the sensor correlation algorithm for illuminant classification and we discuss four changes that improve the algorithm's estimation accuracy and broaden its applicability. First, we space the classification illuminants evenly along the reciprocal scale of color temperature, called "mired," rather than the original color-temperature scale. This improves the perceptual uniformity of the illuminant classification set. Second, we calculate correlation values between the image color gamut and the reference illuminant gamut, rather than between the image pixels and the illuminant gamuts. This change makes the algorithm more reliable. Third, we introduce a new image scaling operation to adjust for overall intensity differences between images. Fourth, we develop the three-dimensional classification algorithms using all three-color channels and compare this with the original two algorithms from the viewpoint of accuracy and computational efficiency. The image processing algorithms incorporating these changes are evaluated using a real image database with calibrated scene illuminants.

In this paper we argue that biological vision and electronic image acquisition share common principles despite their vastly different implementations. These shared principles are based on the need to acquire a common set of input stimuli as well as the need to generalize from the acquired images. Two related principles are discussed in detail, namely, multiple parallel image representations and the use of dedicated local memory in various stages of acquisition and processing. We review relevant literature in visual neuroscience and image systems engineering to support our argument. Particularly, the paper discusses multiple capture image acquisition, with applications such as dynamic range, field-of-view, or depth-of-field extension. Finally, as an example, a novel multiple-capture-single-image complementary metal-oxide-semiconductor sensor is presented. This sensor illustrates the principles that are shared among biological vision and image acquisition.

An effective image reproduction pipeline, spanning image capture, processing and display, must be designed to account for the properties of the human observer. In designing an image pipeline, three principles of human vision are particularly important: trichromacy, color adaptation, and pattern-color sensitivity. These properties also play an important role in metrics used to evaluate image quality reproduction. The main portion of this review comprises a description of these properties of the visual system and how these principles are incorporated into the image reproduction pipeline. The last part of this review describes a new image capture technology, based on a digital pixel fabricated on a CMOS process. This sensor is well-designed for exploring a novel image pipeline architecture that we call multiple-capture, single-image. This architecture is being developed to serve features of human vision that are not yet incorporated in the conventional pipeline.

We describe a system that is being used to segment gray matter and create connected cortical representations from MRI. The method exploits knowledge of the anatomy of the cortex and incorporates structural constraints into the segmentation. First, the white matter and CSF regions in the MR volume are segmented using some novel techniques of posterior anisotropic diffusion. Then, the user selects the cortical white matter component of interest, and its structure is verified by checking for cavities and handles. After this, a connected representation of the gray matter is created by a constrained growing-out from the white matter boundary. Because the connectivity is computed, the segmentation can be used as input to several methods of visualizing the spatial pattern of cortical activity within gray matter. In our case, the connected representation of gray matter is used to create a representation of the flattened cortex. Then, fMRI measurements are overlaid on the flattened representation, yielding a representation of the volumetric data within a single image.

Teo et al. (see IEEE Trans. on Medical Imaging, 1997) proposed an efficient image segmentation technique that anisotropically smoothes the homogeneous posterior probabilities before independent pixel wise MAP classification is carried out. In this paper we develop the mathematical theory underlying the technique. We demonstrate that prior anisotropic smoothing of the posterior probabilities yields the MAP solution of a discrete MRF with a non-interacting, analog discontinuity field. In contrast, isotropic smoothing of the posterior probabilities is equivalent to computing the MAP solution of a single, discrete MRF using continuous relaxation labeling. Combining a discontinuity field with a discrete MRF is important as it allows the disabling of clique potentials across discontinuities. Furthermore, explicit representation of the discontinuity field suggests new algorithms that incorporate properties like hysteresis and non-maximal suppression.

We describe a system that is being used to segment gray matter from magnetic resonance imaging (MRI) and to create connected cortical representations for functional MRI visualization (fMRI). The method exploits knowledge of the anatomy of the cortex and incorporates structural constraints into the segmentation. First, the white matter and cerebral spinal fluid (CSF) regions in the MR volume are segmented using a novel techniques of posterior anisotropic diffusion. Then, the user selects the cortical white matter component of interest, and its structure is verified by checking for cavities and handles. After this, a connected representation of the gray matter is created by a constrained growing-out from the white matter boundary. Because the connectivity is computed, the segmentation can be used as input to several methods of visualizing the spatial pattern of cortical activity within gray matter. In our case, the connected representation of gray matter is used to create a flattened representation of the cortex. Then, fMRI measurements are overlaid on the flattened representation, yielding a representation of the volumetric data within a single image. The software is freely available to the research community.

A new Image Systems Engineering Program (ISEP) has recently been launched at Stanford University. The program includes more than a dozen faculty participants drawn from four departments. The planned stages of growth of the Program are described, with emphasis on the anticipated respective roles of the university and industry.

I describe a method for performing the synthesis and analysis of digital color images. The method is based on two principles. First, image data are represented with respect to the separate physical factors, surface reflectance and the spectral power distribution of the ambient light, that give rise to the perceived color of an object. Second, the encoding is made efficient by using a basis expansion for the surface spectral reflectance and spectral power distribution of the ambient light that takes advantage of the high degree of correlation across the visible wavelengths normally found in such functions. Within this framework, the same basic methods can be used to synthesize image data for color display monitors and printed materials, and to analyze image data into estimates of the spectral power distribution and surface spectral reflectances. The method can be applied to a variety of tasks. Examples of applications include the color balancing of color images and the identification of material surface spectral reflectance when the lighting cannot be completely controlled.