How do high-level visual regions process the temporal aspects of our visual experience? While the temporal sensitivity of early visual cortex has been studied with fMRI in humans, temporal processing in high-level visual cortex is largely unknown. By modeling neural responses with millisecond precision in separate sustained and transient channels, and introducing a flexible encoding framework that captures differences in neural temporal integration time windows and response nonlinearities, we predict fMRI responses across visual cortex for stimuli ranging from 33 ms to 20 s. Using this innovative approach, we discovered that lateral category-selective regions respond to visual transients associated with stimulus onsets and offsets but not sustained visual information. Thus, lateral category-selective regions compute moment-to-moment visual transitions, but not stable features of the visual input. In contrast, ventral category-selective regions process both sustained and transient components of the visual input. Our model revealed that sustained channel responses to prolonged stimuli exhibit adaptation, whereas transient channel responses to stimulus offsets are surprisingly larger than for stimulus onsets. This large offset transient response may reflect a memory trace of the stimulus when it is no longer visible, whereas the onset transient response may reflect rapid processing of new items. Together, these findings reveal previously unconsidered, fundamental temporal mechanisms that distinguish visual streams in the human brain. Importantly, our results underscore the promise of modeling brain responses with millisecond precision to understand the underlying neural computations.

The category of visual stimuli has been reliably decoded from patterns of neural activity in extrastriate visual cortex [1]. It has yet to be seen whether object identity can be inferred from this activity. We present fMRI data measuring responses in human extrastriate cortex to a set of 12 distinct object images. We use a simple winner-take-all classifier, using half the data from each recording session as a training set, to evaluate encoding of object identity across fMRI voxels. Since this approach is sensitive to the inclusion of noisy voxels, we describe two methods for identifying subsets of voxels in the data which optimally distinguish object identity. One method characterizes the reliability of each voxel within subsets of the data, while another estimates the mutual information of each voxel with the stimulus set. We find that both metrics can identify subsets of the data which reliably encode object identity, even when noisy measurements are artificially added to the data. The mutual information metric is less efficient at this task, likely due to constraints in fMRI data.

The maximization of diversity of neuronal response properties has been recently suggested as an organizing principle for the formation of such prominent features of the functional architecture of the brain as the corti(cid:173) cal columns and the associated patchy projection patterns (Malach, 1994). We show that (1) maximal diversity is attained when the ratio of dendritic and axonal arbor sizes is equal to one, as found in many cortical areas and across species (Lund et al., 1993; Malach, 1994), and (2) that maxi(cid:173) mization of diversity leads to better performance in systems of receptive fields implementing steerable/shiftable filters, and in matching spatially distributed signals, a problem that arises in many high-level visual tasks. 1 Anatomical substrate for sampling diversity A fundamental feature of cortical architecture is its columnar organization, mani(cid:173) fested in the tendency of neurons with similar properties to be organized in columns that run perpendicular to the cortical surface. This organization of the cortex was ini(cid:173) tially discovered by physiological experiments (Mouncastle, 1957; Hubel and Wiesel, 1962), and subsequently confirmed with the demonstration of histologically defined columns. Tracing experiments have shown that axonal projections throughout the cerebral cortex tend to be organized in vertically aligned clusters or patches. In par(cid:173) ticular, intrinsic horizontal connections linking neighboring cortical sites, which may extend up to 2 - 3 mm, have a striking tendency to arborize selectively in preferred sites, forming distinct axonal patches 200 - 300 J.lm in diameter. Recently, it has been observed that the size of these patches matches closely the average diameter of individual dendritic arbors of upper-layer pyramidal cells 118 Kalanit Grill Spector, Shimon Edelman, Rafael Malach