Burlingham CS, Mirbagheri S, Heeger DJ, A unified model of the task-evoked pupil response, Science Advances, 8:eabi9979 DOI: 10.1126/sciadv.abi9979, 2022.
Abstract: The pupil dilates and reconstricts following task events. It is popular to model this task-evoked pupil response as a linear transformation of event-locked impulses, whose amplitudes are used as estimates of arousal. We show that this model is incorrect and propose an alternative model based on the physiological finding that a common neural input drives saccades and pupil size. The estimates of arousal from our model agreed with key predictions: Arousal scaled with task difficulty and behavioral performance but was invariant to small differences in trial duration. Moreover, the model offers a unified explanation for a wide range of phenomena: entrainment of pupil size and saccades to task timing, modulation of pupil response amplitude and noise with task difficulty, reaction time dependent modulation of pupil response timing and amplitude, a constrictory pupil response time-locked to saccades, and task-dependent distortion of this saccade-locked pupil response.
Denison RN, Carrasco M, Heeger DJ, A dynamic normalization model of temporal attention, Nature Human Behavior, 5:1674-1685, 2021.
Abstract: Vision is dynamic, handling a continuously changing stream of input, yet most models of visual attention are static. Here, we develop a dynamic normalization model of visual temporal attention and constrain it with new psychophysical human data. We manipulated temporal attention—the prioritization of visual information at specific points in time—to a sequence of two stimuli separated by a variable time interval. Voluntary temporal attention improved perceptual sensitivity only over a specific interval range. To explain these data, we modelled voluntary and involuntary attentional gain dynamics. Voluntary gain enhancement took the form of a limited resource over short time intervals, which recovered over time. Taken together, our theoretical and experimental results formalize and generalize the idea of limited attentional resources across space at a single moment to limited resources across time at a single location.
Jigo M, Heeger DJ, Carrasco M, An image-computable model on how endogenous and exogenous attention differentially alter visual perception, Proc. Nat'l Acad. Sci. USA, 118 (33) e2106436118; DOI: 10.1073/pnas.2106436118, 2021.
Abstract: Attention alters perception across the visual field. Typically, endogenous (voluntary) and exogenous (involuntary) attention similarly improve performance in many visual tasks, but they have differential effects in some tasks. Extant models of visual attention assume that the effects of these two types of attention are identical and consequently do not explain differences between them. Here, we develop a model of spatial resolution and attention that distinguishes between endogenous and exogenous attention. We focus on texture-based segmentation as a model system because it has revealed a clear dissociation between both attention types. For a texture for which performance peaks at parafoveal locations, endogenous attention improves performance across eccentricity, whereas exogenous attention improves performance where the resolution is low (peripheral locations) but impairs it where the resolution is high (foveal locations) for the scale of the texture. Our model emulates sensory encoding to segment figures from their background and predict behavioral performance. To explain attentional effects, endogenous and exogenous attention require separate operating regimes across visual detail (spatial frequency). Our model reproduces behavioral performance across several experiments and simultaneously resolves three unexplained phenomena: 1) the parafoveal advantage in segmentation, 2) the uniform improvements across eccentricity by endogenous attention, and 3) the peripheral improvements and foveal impairments by exogenous attention. Overall, we unveil a computational dissociation between each attention type and provide a generalizable framework for predicting their effects on perception across the visual field.
Listman JB, Tsay J, Kim HE, Mackey WE, Heeger DJ, Long-term Motor Learning in the Wild with High Volume Video Game Data, Frontiers in Human Neuroscience, DOI: 10.3389/fnhum.2021.777779, 2021.
Abstract: Motor learning occurs over long periods of practice during which motor acuity, the ability to execute actions more accurately, precisely, and in less time, improves. Laboratory-based studies of motor learning are typically limited to a small number of participants and a time frame of minutes to several hours per participant. There is a need to assess the generalizability of theories and findings from lab-based motor learning studies on larger samples and time scales. In addition, laboratory-based studies of motor learning use relatively simple motor tasks which participants are unlikely to be intrinsically motivated to learn, limiting the interpretation of their findings in more ecologically valid settings (''in the wild''). We studied the acquisition and longitudinal refinement of a complex sensorimotor skill embodied in a first-person shooter video game scenario, with a large sample size (N = 7174, 682,564 repeats of the 60 s game) over a period of months. Participants voluntarily practiced the gaming scenario for up to several hours per day up to 100 days. We found improvement in performance accuracy (quantified as hit rate) was modest over time but motor acuity (quantified as hits per second) improved considerably, with 40-60% retention from 1 day to the next. We observed steady improvements in motor acuity across multiple days of video game practice, unlike most motor learning tasks studied in the lab that hit a performance ceiling rather quickly. Learning rate was a non-linear function of baseline performance level, amount of daily practice, and to a lesser extent, number of days between practice sessions. In addition, we found that the benefit of additional practice on any given day was non- monotonic; the greatest improvements in motor acuity were evident with about an hour of practice and 90% of the learning benefit was achieved by practicing 30 min per day. Taken together, these results provide a proof-of-concept in studying motor skill acquisition outside the confines of the traditional laboratory, in the presence of unmeasured confounds, and provide new insights into how a complex motor skill is acquired in an ecologically valid setting and refined across much longer time scales than typically explored.
Heeger DJ, Zemlianova KO, A recurrent circuit implements normalization, simulating the dynamics of V1 activity, Proc. Nat'l Acad. Sci. USA, 117:22494-22505, 2020.
Abstract: The normalization model has been applied to explain neural activity in diverse neural systems including primary visual cortex (V1). The model's defining characteristic is that the response of each neuron is divided by a factor that includes a weighted sum of activity of a pool of neurons. Despite the success of the normalization model, there are three unresolved issues. 1) Experimental evidence supports the hypothesis that normalization in V1 operates via recurrent amplification, i.e., amplifying weak inputs more than strong inputs. It is unknown how normalization arises from recurrent amplification. 2) Experiments have demonstrated that normalization is weighted such that each weight specifies how one neuron contributes to another’s normalization pool. It is unknown how weighted normalization arises from a recurrent circuit. 3) Neural activity in V1 exhibits complex dynamics, including gamma oscillations, linked to normalization. It is unknown how these dynamics emerge from normalization. Here, a family of recurrent circuit models is reported, each of which comprises coupled neural integrators to implement normalization via recurrent amplification with arbitrary normalization weights, some of which can recapitulate key experimental observations of the dynamics of neural activity in V1.
Burlingham CS, Heeger DJ, Heading perception depends on time-varying evolution of optic flow, Proc. Nat'l Acad. Sci. USA, 117:33161-33169, 2020..
Abstract: There is considerable support for the hypothesis that perception of heading in the presence of rotation is mediated by instantaneous optic flow. This hypothesis, however, has never been tested. We introduce a method, termed ''nonvarying phase motion'', for generating a stimulus that conveys a single instantaneous optic flow field, even though the stimulus is presented for an extended period of time. In this experiment, observers viewed stimulus videos and performed a forced-choice heading discrimination task. For nonvarying phase motion, observers made large errors in heading judgments. This suggests that instantaneous optic flow is insufficient for heading perception in the presence of rotation. These errors were mostly eliminated when the velocity of phase motion was varied over time to convey the evolving sequence of optic flow fields corresponding to a particular heading. This demon- strates that heading perception in the presence of rotation relies on the time-varying evolution of optic flow. We hypothesize that the visual system accurately computes heading, despite rotation, based on optic acceleration, the temporal derivative of optic flow.
Heeger DJ, Mackey, WE, Oscillatory Recurrent Gated Neural Integrator Circuits (ORGaNICs), a unifying theoretical framework for neural dynamics, Proc. Nat'l Acad. Sci. USA, 116:22783-22794, 2019.
Abstract: Working memory is an example of a cognitive and neural process that is not static but evolves dynamically with changing sensory inputs; another example is motor preparation and execution. We introduce a theoretical framework for neural dynamics, based on oscillatory recurrent gated neural integrator circuits (ORGaNICs), and apply it to simulate key phenomena of working memory and motor control. The model circuits simulate neural activity with complex dynamics, including sequential activity and traveling waves of activity, that manipulate (as well as maintain) information during working memory. The same circuits convert spatial patterns of premotor activity to temporal profiles of motor control activity and manipulate (e.g., time warp) the dynamics. Derivative-like recurrent connectivity, in particular, serves to manipulate and update internal models, an essential feature of working memory and motor execution. In addition, these circuits incorporate recurrent normalization, to ensure stability over time and robustness with respect to perturbations of synaptic weights.
McClain K, Tingley D, Heeger DJ, Buzsaki G, Position-theta-phase model of hippocampal place cell activity applied ot quantification of running speed modulation of firing rate, Proc. Nat'l Acad. Sci. USA, 116:27035-27042, 2019.
Abstract: Spiking activity of place cells in the hippocampus encodes the animal's position as it moves through an environment. Within a cell's place field, both the firing rate and the phase of spiking in the local theta oscillation contain spatial information. We propose a position–theta-phase (PTP) model that captures the simultaneous expression of the firing-rate code and theta-phase code in place cell spiking. This model parametrically characterizes place fields to compare across cells, time, and conditions; generates realistic place cell simulation data; and conceptualizes a framework for principled hy- pothesis testing to identify additional features of place cell activity. We use the PTP model to assess the effect of running speed in place cell data recorded from rats running on linear tracks. For the majority of place fields, we do not find evidence for speed modulation of the firing rate. For a small subset of place fields, we find firing rates significantly increase or decrease with speed. We use the PTP model to compare candidate mechanisms of speed modulation in significantly modulated fields and determine that speed acts as a gain control on the magnitude of firing rate. Our model provides a tool that connects rigorous analysis with a computational framework for understanding place cell activity.
Roth, ZN, Heeger DJ, Merriam E, Stimulus vignetting and orientation selectivity in human visual cortex, eLife, 7:e37241 DOI: 10.7554/eLife.37241, 2018.
Abstract: Neural selectivity to orientation is one of the simplest and most thoroughly-studied cortical sensory features. Here, we show that a large body of research that purported to measure orientation tuning may have in fact been inadvertently measuring sensitivity to second-order changes in luminance, a phenomenon we term 'vignetting'. Using a computational model of neural responses in primary visual cortex (V1), we demonstrate the impact of vignetting on simulated V1 responses. We then used the model to generate a set of predictions, which we confirmed with functional MRI experiments in human observers. Our results demonstrate that stimulus vignetting can wholly determine the orientation selectivity of responses in visual cortex measured at a macroscopic scale, and suggest a reinterpretation of a well-established literature on orientation processing in visual cortex.
Heeger DJ, Theory of Cortical Function, Proc Natl Acad Sci USA, 114:1773-1782, 2017.
Abstract: Most models of sensory processing in the brain have a feedforward architecture in which each stage comprises simple linear filtering operations and nonlinearities. Models of this form have been used to explain a wide range of neurophysiological and psychophysical data, and many recent successes in artificial intelligence (with deep convolutional neural nets) are based on this architecture. But neocortex is not a feedforward architecture. This paper proposes a first step toward an alternative computational framework in which neural activity in each brain area depends on a combination of feedforward drive (bottom-up from the previous processing stage), feedback drive (top-down context from the next stage), and prior drive (expectation). The relative contributions of feedforward drive, feedback drive, and prior drive are controlled by a handful of state parameters, which I hypothesize correspond to neuromodulators and oscillatory activity. In some states, neural responses are dominated by the feedforward drive and the theory is identical to a conventional feedforward model, thereby preserving all the desirable features of those models. In other states, the theory is a generative model that constructs a sensory representation from an abstract representation, like memory recall. In still other states, the theory combines prior expectation with sensory input, explores different possible perceptual interpretations of ambiguous sensory inputs, and predicts forward in time. The theory, therefore, offers an empirically-testable framework for understanding how the cortex accomplishes inference, exploration, and prediction.
Li HH, Rankin J, Rinzel J, Carrasco M, Heeger DJ, Attention Model of Binocular Rivalry, Proc Natl Acad Sci USA, E6192-E6201, 2017.
Abstract: When the corresponding retinal locations in the two eyes are presented with incompatible images, a stable percept gives way to perceptual alternations in which the two images compete for perceptual dominance. As perceptual experience evolves dynami- cally under constant external inputs, binocular rivalry has been used for studying intrinsic cortical computations and for understanding how the brain regulates competing inputs. Converging behavioral and EEG results have shown that binocular rivalry and attention are intertwined: binocular rivalry ceases when attention is diverted away from the rivalry stimuli. In addition, the competing image in one eye suppresses the target in the other eye through a pattern of gain changes similar to those induced by attention. These results require a revision of the current computational theories of binocular rivalry, in which the role of attention is ignored. Here, we provide a computational model of binocular rivalry. In the model, competition between two images in rivalry is driven by both attentional modu- lation and mutual inhibition, which have distinct selectivity (feature vs. eye of origin) and dynamics (relatively slow vs. relatively fast). The proposed model explains a wide range of phenomena reported in rivalry, including the three hallmarks: (i) binocular rivalry requires attention; (ii) various perceptual states emerge when the two im- ages are swapped between the eyes multiple times per second; (iii) the dominance duration as a function of input strength follows Levelt’s propositions. With a bifurcation analysis, we identified the parameter space in which the model’s behavior was consistent with experimental results.
Westrick JM, Heeger DJ, Landy MS, Pattern adaptation and normalization reweighting, Journal of Neuroscience, 36:9805-9816, 2016.
Abstract: Adaptation to an oriented stimulus changes both the gain and preferred orientation of neural responses in V1. Neurons tuned near the adapted orientation are suppressed, and their preferred orientations shift away from the adapter. We propose a model in which weights of divisive normalization are dynamically adjusted to homeostatically maintain response products between pairs of neurons. We demonstrate that this adjustment can be performed by a very simple learning rule. Simulations of this model closely match existing data from visual adaptation experiments. We consider several alternative models, including variants based on homeostatic maintenance of re- sponse correlations or covariance, as well as feedforward gain-control models with multiple layers, and we demonstrate that homeostatic maintenance of response products provides the best account of the physiological data.
Haigh SM, Gupta AG, Barb SM, Glass SAF, Minshew NJ, Dinstein I, Heeger DJ, Eack SM, Behrmann M, Differential sensory fMRI signatures in autism and schizophrenia: analysis of amplitude and trial-to-trial variability, Schizophrenia Research, 175:12-19, 2016.
Abstract: Autism and schizophrenia share multiple phenotypic and genotypic markers, and there is ongoing debate regarding the relationship of these two disorders. To examine whether cortical dynamics are similar across these disorders, we directly compared fMRI responses to visual, somatosensory and auditory stimuli in adults with autism (N = 15), with schizophrenia (N = 15), and matched controls (N = 15). All participants completed a one-back letter detection task presented at fixation (to control attention) while task-irrelevant sensory stimulation was delivered to the different modalities. We focused specifically on the response amplitudes and the variability in sensory fMRI responses of the two groups, given the evidence of greater trial-to-trial variability in adults with autism. Both autism and schizophrenia individuals showed weaker signal-to-noise ratios (SNR) in sensory-evoked responses compared to controls (d > 0.42), but for different reasons. For the autism group, the fMRI response amplitudes were indistinguishable from controls but were more variable trial-to-trial (d = 0.47). For the schizophrenia group, response amplitudes were smaller compared to autism (d = 0.44) and control groups (d = 0.74), but were not significantly more variable (d < 0.29). These differential group profiles suggest (1) that greater trial-to-trial variability in cortical responses may be specific to autism and is not a defining characteristic of schizophrenia, and (2) that blunted response amplitudes may be characteristic of schizophrenia. The relationship between the amplitude and the variability of cortical activity might serve as a specific signature differentiating these neurodevelopmental disorders. Identifying the neural basis of these responses and their relationship to the underlying genetic bases may substantially enlighten the understanding of both disorders.
Haigh SM, Heeger DJ, Heller L, Gupta A, Dinstein I, Minshew NJ, Behrmann M, No difference in cross-modal attention or sensory discrimination thresholds in autism and matched controls, Vision Research, 121:85-94, 2016.
Abstract: Autism has been associated with abnormalities in sensory and attentional processing. Here, we assessed these processes independently in the visual and auditory domains using a visual contrast-discrimination task and an auditory modulation-depth discrimination task. To evaluate changes in sensory function by attention, we measured behavioral performance (discrimination accuracy) when subjects were cued to attend and respond to the same stimulus (frequent valid cue) or cued to attend to one stimulus and respond to the non-cued stimulus (infrequent invalid cue). The stimuli were presented at threshold to ensure equal difficulty across participants and groups. Results from fifteen high-functioning adult individuals with autism and fifteen matched controls revealed no significant differences in visual or auditory discrimination thresholds across groups. Furthermore, attention robustly modulated performance accuracy (performance was better for valid than invalid cues) in both sensory modalities and to an equivalent extent in both groups. In conclusion, when using this well-controlled method, we found no evidence of atypical sensory function or atypical attentional modulation in a group of high functioning individuals with clear autism symptomatology.
Wang HX, Yuval-Greenberg S, Heeger DJ, Suppressive interactions underlying visually evoked fixational saccades, Vision Research, 118:70-82, 2016.
Abstract: Small saccades occur frequently during fixation, and are coupled to changes in visual stimulation and cognitive state. Neurophysiologically, fixational saccades reflect neural activity near the foveal region of a continuous visuomotor map. It is well known that competitive interactions between neurons within visuomotor maps contribute to target selection for large saccades. Here we asked how such interactions in visuomotor maps shape the rate and direction of small fixational saccades. We measured fixational saccades during periods of prolonged fixation while presenting pairs of visual stimuli (parafoveal: 0.8° eccentricity; peripheral: 5° eccentricity) of various contrasts. Fixational saccade direction was biased toward locations of parafoveal stimuli but not peripheral stimuli, ~100–250 ms following stimulus onset. The rate of fixational saccades toward parafoveal stimuli (congruent saccades) increased systematically with parafoveal stimulus contrast, and was suppressed by the simultaneous presentation of a peripheral stimulus. The suppression was best characterized as a combination of two processes: a subtractive suppression of the overall fixational saccade rate and a divisive suppression of the direction bias. These results reveal the nature of suppressive interactions within visuomotor maps and constrain models of the population code for fixational saccades.
Haigh SM, Minshew N, Heeger DJ, Dinstein I, Behrmann, M, Over-responsiveness and greater variability in roughness perception in autism, Autism Research, 9:393-402, 2016.
Abstract: Although sensory problems, including tactile hypersensitivity and hyposensitivity (DSM‐5) are commonly associated with autism, there is a dearth of systematic and rigorous research in this domain. Here, we report findings from a psychophysical experiment that explored differences in tactile perception between individuals with autism and typically developing control participants, who, using their index finger, rated a series of surfaces on the extent of their roughness. Each surface was rated multiple times and we calculated both the average rating and the variability across trials. Relative to controls, the individuals with autism perceived the surfaces as rougher overall and exhibited greater variability in their ratings across trials. These findings characterize altered tactile perception in autism and suggest that sensory problems in autism may be the product of overly responsive and variable sensory processing.
Franchak JM, Heeger DJ, Hasson U, Adolph KE, Free viewing in infants and adults, Infancy, 21:262-287, 2015.
Abstract: The current study investigated age differences in free viewing gaze behavior. Adults and 6-, 9-, 12-, and 24-month-old infants watched a 60-sec Sesame Street video clip while their eye movements were recorded. Adults displayed high intersubject consistency in eye movements; they tended to fixate the same places at the same. Infants showed weaker consistency between observers and intersubject consistency increased with age. Across age groups, the influence of both bottom-up features (fixating visually salient areas) and top-down features (looking at faces) increased. Moreover, individual differences in fixating bottom-up and top-down features predicted whether infants’ eye movements were consistent with those of adults, even when controlling for age. However, this relation was moderated by the number of faces available in the scene, suggesting that the development of adult-like viewing involves learning when to prioritize looking at bottom-up and top- down features.
Brouwer GJ, Arnedo V, Offen S, Heeger DJ, Grant A, Normalization in human somatosensory cortex, Journal of Neurophysiology, 114:2588-2599, 2015.
Abstract: Cross-orientation suppression was measured in human primary visual cortex (V1) to test the normalization model. Subjects viewed vertical target gratings (of varying contrasts) with or without a superimposed horizontal mask grating (fixed contrast). We used functional magnetic resonance imaging (fMRI) to measure the activity in each of several hypothetical channels (corresponding to subpopulations of neurons) with different orientation tunings and fit these orientation-selective responses with the normalization model. For the V1 channel maximally tuned to the target orientation, responses increased with target contrast but were suppressed when the horizontal mask was added, evident as a shift in the contrast gain of this channel’s responses. For the channel maximally tuned to the mask orientation, a constant baseline response was evoked for all target contrasts when the mask was absent; responses decreased with increasing target contrast when the mask was present. The normalization model provided a good fit to the contrast-response functions with and without the mask. In a control experiment, the target and mask presentations were temporally interleaved, and we found no shift in contrast gain, i.e., no evidence for suppression. We conclude that the normalization model can explain cross-orientation suppression in human visual cortex. The approach adopted here can be applied broadly to infer, simultaneously, the responses of several subpopulations of neurons in the human brain that span particular stimulus or feature spaces, and characterize their interactions. In addition, it allows us to investigate how stimuli are represented by the inferred activity of entire neural populations.
Harris H, Israeli D, Minshew N, Bonneh Y, Heeger DJ, Behrmann M, Sagi S, Perceptual learning in autism: over-specificity and possible remedies, Nature Neuroscience, 18:1574-1576, 2015.
Abstract: Inflexible behavior is a core characteristic of autism spectrum disorder (ASD), but its underlying cause is unknown. Using a perceptual learning protocol, we observed initially efficient learning in ASD that was followed by anomalously poor learning when the location of the target was changed (over-specificity). Reducing stimulus repetition eliminated over-specificity. Our results indicate that inflexible behavior may be evident ubiquitously in ASD, even in sensory learning, but can be circumvented by specifically designed stimulation protocols.
Li HH, Carasco M, Heeger DJ, Deconstruction interocular suppression: Attention and divisive normalization, PLOS Computational Biology, DOI: 10.1371/journal.pcbi.1004510, 2015.
Abstract: In interocular suppression, a suprathreshold monocular target can be rendered invisible by a salient competitor stimulus presented in the other eye. Despite decades of research on interocular suppression and related phenomena (e.g., binocular rivalry, flash suppression, continuous flash suppression), the neural processing underlying interocular suppression is still unknown. We developed and tested a computational model of interocular suppression. The model included two processes that contributed to the strength of interocular suppression: divisive normalization and attentional modulation. According to the model, the salient competitor induced a stimulus-driven attentional modulation selective for the location and orientation of the competitor, thereby increasing the gain of neural responses to the competitor and reducing the gain of neural responses to the target. Additional suppression was induced by divisive normalization in the model, similar to other forms of visual masking. To test the model, we conducted psychophysics experiments in which both the size and the eye-of-origin of the competitor were manipulated. For small and medium competitors, behavioral performance was consonant with a change in the response gain of neurons that responded to the target. But large competitors induced a contrast-gain change, even when the competitor was split between the two eyes. The model correctly predicted these results and outperformed an alternative model in which the attentional modulation was eye specific. We conclude that both stimulus-driven attention (selective for location and feature) and divisive normalization contribute to interocular suppression.
Farbood MM, Heeger DJ, Marcus G, Hasson U, Lerner Y, The neural processing of hierarchical structure in music and speech at different timescales, Frontiers in Neuroscience, in press
Abstract: Music, like speech, is a complex auditory signal that contains structures at multiple timescales, and as such a potentially powerful entry point into the question of how the brain integrates complex streams of information. Using an experimental design modeled after previous studies that used scrambled versions of a spoken story (Lerner et al., 2011) and a silent movie (Hasson et al., 2008), we investigate whether listeners perceive hierarchical structure in music beyond short (~6 sec) time windows and whether there is cortical overlap between music and language processing at multiple timescales. Experienced pianists were presented with an extended musical excerpt scrambled at multiple timescales - by measure, phrase, and section - while measuring brain activity with functional magnetic resonance imaging (fMRI). The reliability of evoked activity, as quantified by inter-subject correlation of the fMRI responses was measured. We found that response reliability depended systematically on musical structural coherence, revealing a topographically organized hierarchy of processing timescales. Early auditory areas (at the bottom of the hierarchy) responded reliably in all conditions. For brain areas at the top of the hierarchy, the original (unscrambled) excerpt evoked more reliable responses than any of the scrambled excerpts, indicating that these brain areas process long-timescale musical structures, on the order of minutes. The topography of processing timescales was analogous with that reported previously for speech, but the timescale gradients for music and speech overlapped with one another only partially, suggesting that temporally analogous structure - words/measures, sentences/musical phrases, paragraph/sections - are processed separately.
Dinstein I, Behrmann M, Heeger DJ, Neural variability: Friend or Foe? Trends in Cognitive Sciences, in press
Abstract: Although we may not realize it, our brain function varies markedly from moment to moment such that our brain responses exhibit substantial variability across trials even in response to a simple repeating stimulus. Should we care about such within‐subject variability? Are there developmental, cognitive, and clinical consequences to having a brain that is more or less variable/noisy? Although neural variability seems to be beneficial for learning, excessive levels of neural variability are apparent in individuals with different clinical disorders. We propose that measuring distinct types of neural variability in autism and other disorders is likely to reveal critical insights regarding the neuropathology of the different disorders. We further discuss the importance of studying neural variability more generally across development and aging in humans.
Haigh SM, Heeger DJ, Dinstein I, Minshew N, Behrmann, M, Cortical variability in the sensory-evoked response in autism, Journal of Austism and Developmental Disorders, 45:1176-1190, 2015
Abstract: Previous findings have shown that individuals with autism spectrum disorder (ASD) evince greater intraindividual variability (IIV) in their sensory-evoked fMRI responses compared to typical control participants. We explore the robustness of this finding with a new sample of high-functioning adults with autism. Participants were presented with visual, somatosensory and auditory stimuli in the scanner whilst they completed a one-back task. While ASD and control participants were statistically indistinguishable with respect to behavioral responses, the new ASD group exhibited greater IIV relative to controls. We also show that the IIV was equivalent across hemispheres and remained stable over the duration of the experiment. This suggests that greater cortical IIV may be a replicable characteristic of sensory systems in autism.
Grubb MA, White AL, Heeger DJ, Carrasco M, Interactions between voluntary and involuntary attention modulate the quality and temporal dynamics of visual processing, Psychonomic Bulletin & Review, 22:437-444, 2015
Abstract: Successfully navigating a dynamic environment requires the efficient distribution of finite neural resources. Voluntary (endogenous) covert spatial attention selectively allocates those processing resources to goal-relevant locations in the visual scene in the absence of eye movements. However, the allocation of spatial attention is not always voluntary; abrupt onsets in the visual periphery automatically enhance processing of nearby stimuli (exogenous attention). In dynamic environments, exogenous events and internal goals likely compete to determine the distribution of attention, but how such competition is resolved is not well understood. To investigate how exogenous events interact with the concurrent allocation of voluntary attention, we used a speed– accuracy trade-off (SAT) procedure. SAT conjointly measures the rate of information accrual and asymptotic discriminability, allowing us to measure how attentional interactions unfold over time during stimulus processing. We found that both types of attention sped information accrual and improved discriminability. However, focusing endogenous attention at the target location reduced the effects of exogenous cues on the rate of information accrual and rendered negligible their effects on asymptotic discriminability. We verified the robustness of these findings in four additional experiments that targeted specific, critical response delays. In conclusion, the speed and quality of visual processing depend conjointly on internally and externally driven attentional states, but it is possible to voluntarily diminish distraction by irrelevant events in the periphery.
Yang Z, Heeger DJ, Blake R, Seidemann E, Long-range traveling waves of activity triggered by local dichoptic stimulation in V1 of behaving monkeys, Journal of Neurophysiology, 113:277-294, 2015
Abstract: Traveling waves of cortical activity, in which local stimulation triggers lateral spread of activity to distal locations, have been hypothesized to play an important role in cortical function. However, there is conflicting physiological evidence for the existence of spreading traveling waves of neural activity triggered locally. Dichoptic stimulation, in which the two eyes view dissimilar monocular patterns, can lead to dynamic wave-like fluctuations in visual perception and therefore, provides a promising means for identifying and studying cortical traveling waves. Here, we used voltage-sensitive dye imaging to test for the existence of traveling waves of activity in the primary visual cortex of awake, fixating monkeys viewing dichoptic stimuli. We find clear traveling waves that are initiated by brief, localized contrast increments in one of the monocular patterns and then, propagate at speeds of ~30 mm/s. These results demonstrate that under an appropriate visual context, circuitry in visual cortex in alert animals is capable of supporting long-range traveling waves triggered by local stimulation.
Wang HX, Merriam EP, Freeman F, Heeger DJ, Motion direction biases and decoding in human visual cortex, Journal of Neuroscience, 34:12601-12615, 2014
Abstract: Functional magnetic resonance imaging (fMRI) studies have relied on multivariate analysis methods to decode visual motion direction from measurements of cortical activity. Above-chance decoding has been commonly used to infer the motion-selective response properties of the underlying neural populations. Moreover, patterns of reliable response biases across voxels that underlie decoding have been interpreted to reflect maps of functional architecture. Using fMRI, we identified a direction-selective response bias in human visual cortex that: (1) predicted motion-decoding accuracy; (2) depended on the shape of the stimulus aperture rather than the absolute direction of motion, such that response amplitudes gradually decreased with distance from the stimulus aperture edge corresponding to motion origin; and 3) was present in V1, V2, V3, but not evident in MT+, explaining the higher motion-decoding accuracies reported previously in early visual cortex. These results demonstrate that fMRI-based motion decoding has little or no dependence on the underlying functional organization of motion selectivity.
Yuval-Greenberg S, Merriam EP, Heeger DJ, Spontaneous microsaccades reflect shifts in covert attention, Journal of Neuroscience, 34:13693-13700, 2014
Abstract: Microsaccade rate during fixation is modulated by the presentation of a visual stimulus. When the stimulus is an endogenous attention cue, the ensuing microsaccades tend to be directed toward the cue. This finding has been taken as evidence that microsaccades index the locus of spatial attention. But the vast majority of microsaccades that subjects make are not triggered by visual stimuli. Under natural viewing conditions, spontaneous microsaccades occur frequently (2-3 Hz), even in the absence of a stimulus or a task. While spontaneous microsaccades may depend on low-level visual demands, such as retinal fatigue, image fading, or fixation shifts, it is unknown whether their occurrence corresponds to changes in the attentional state. We developed a protocol to measure whether spontaneous microsaccades reflect shifts in spatial attention. Human subjects fixated a cross while microsaccades were detected from streaming eye-position data. Detection of a microsaccade triggered the appearance of a peripheral ring of grating patches, which were followed by an arrow (a postcue) indicating one of them as the target. The target was either congruent or incongruent (opposite) with respect to the direction of the microsaccade (which preceded the stimulus). Subjects reported the tilt of the target (clockwise or counterclockwise relative to vertical). We found that accuracy was higher for congruent than for incongruent trials. We conclude that the direction of spontaneous microsaccades is inherently linked to shifts in spatial attention.
Cutrone CK, Heeger DJ, Carrasco, M, Attention enhances contrast appearance via increased baseline of neural responses, Journal of Vision, 14(14):16, 1-14, 2014
Abstract: Covert spatial attention increases the perceived contrast of stimuli at attended locations, presumably via enhancement of visual neural responses. However, the relation between perceived contrast and the underlying neural responses has not been characterized. In this study, we systematically varied stimulus contrast, using a two-alternative, forced- choice comparison task to probe the effect of attention on appearance across the contrast range. We modeled performance in the task as a function of underlying neural contrast-response functions. Fitting this model to the observed data revealed that an increased input baseline in the neural responses accounted for the enhancement of apparent contrast with spatial attention.
Bonneh, YS, Donner TH, Cooperman A, Heeger DJ, Sagi D, Motion-induced blindness and Troxler fading: common and different mechanisms, PLoS One, 9(3):e92894, 1-8, 2014
Abstract: Extended stabilization of gaze leads to disappearance of dim visual targets presented peripherally. This phenomenon, known as Troxler fading, is thought to result from neuronal adaptation. Intense targets also disappear intermittently when surrounded by a moving pattern (the mask), a phenomenon known as motion-induced blindness (MIB). The similar phenomenology and dynamics of these disappearances may suggest that also MIB is, likewise, solely due to adaptation, which may be amplified by the presence of the mask. Here we directly compared the dependence of both phenomena on target contrast. Observers reported the disappearance and reappearance of a target of varying intensity (contrast levels: 8%-80%). MIB was induced by adding a mask that moved at one of various different speeds. The results revealed a lawful effect of contrast in both MIB and Troxler fading, but with opposite trends. Increasing target contrast increased (doubled) the rate of disappearance events for MIB, but decreased the disappearance rate to half in Troxler fading. The target mean invisible period decreased equally strongly with target contrast in MIB and in Troxler fading. The results suggest that both MIB and Troxler are equally affected by contrast adaptation, but that the rate of MIB is governed by an additional mechanism, possibly involving antagonistic processes between neuronal populations processing target and mask. Our results link MIB to other bi-stable visual phenomena that involve neuronal competition (such as binocular rivalry), which exhibit an analogous dependency on the strength of the competing stimulus components.
Blake R, Brascamp J, Heeger DJ, Can binocular rivalry reveal neural correlates of consciousness?, Philosophical Transactions of the Royal Society B, 369: 20130211, 2014
Abstract: This essay critically examines the extent to which binocular rivalry can provide important clues about the neural correlates of conscious visual perception. Our ideas are presented within the framework of four questions about the use of rivalry for this purpose: (i) what constitutes an adequate comparison condition for gauging rivalry’s impact on awareness, (ii) how can one distinguish abolished awareness from inattention, (iii) when one obtains unequivocal evidence for a causal link between a fluctuating measure of neural activity and fluctuating perceptual states during rivalry, will it generalize to other stimulus conditions and perceptual phenomena and (iv) does such evidence necessarily indicate that this neural activity constitutes a neural correlate of consciousness? While arriving at sceptical answers to these four questions, the essay nonetheless offers some ideas about how a more nuanced utilization of binocular rivalry may still provide fundamental insights about neural dynamics, and glimpses of at least some of the ingredients comprising neural correlates of consciousness, including those involved in perceptual decision-making.
Freeman J, Heeger DJ, Merriam EP, Coarse-scale biases for spirals and orientation in human visual cortex, Journal of Neuroscience, 33:19695-19703, 2013.
Abstract: Multivariate decoding analyses are widely applied to functional magnetic resonance imaging (fMRI) data, but there is controversy over their interpretation. Orientation decoding in primary visual cortex (V1) reflects coarse-scale biases, including an over-representation of radial orientations. But fMRI responses to clockwise and counter-clockwise spirals can also be decoded. Because these stimuli are matched for radial orientation, while differing in local orientation, it has been argued that fine-scale columnar selectivity for orientation contributes to orientation decoding. We measured fMRI responses in human V1 to both oriented gratings and spirals. Responses to oriented gratings exhibited a complex topography, including a radial bias that was most pronounced in the peripheral representation, and a near-vertical bias that was most pronounced near the foveal representation. Responses to clockwise and counter-clockwise spirals also exhibited coarse-scale organization, at the scale of entire visual quadrants. The preference of each voxel for clockwise or counter- clockwise spirals was predicted from the preferences of that voxel for orientation and spatial position (i.e., within the retinotopic map). Our results demonstrate a bias for local stimulus orientation that has a coarse spatial scale, is robust across stimulus classes (spirals and gratings), and suffices to explain decoding from fMRI responses in V1.
Grubb MA, Behrmann M, Egan R, Minshew NJ, Heeger DJ, Carrasco M, Exogenous spatial attention: Evidence for intact functioning in adults with autism spectrum disorder, Journal of Vision, 13(14):9, 1-13, 2013.
Abstract: Deficits or atypicalities in attention have been reported in individuals with autism spectrum disorder (ASD), yet no consensus on the nature of these deficits has emerged. We conducted three experiments that paired a peripheral precue with a covert discrimination task, using protocols for which the effects of covert exogenous spatial attention on early vision have been well established in typically developing populations. Experiment 1 assessed changes in contrast sensitivity, using orientation discrimination of a contrast-defined grating; Experiment 2 evaluated the reduction of crowding in the visual periphery, using discrimination of a letter-like figure with flanking stimuli at variable distances; and Experiment 3 assessed improvements in visual search, using discrimination of the same letter-like figure with a variable number of distractor elements. In all three experiments, we found that exogenous attention modulated visual discriminability in a group of high-functioning adults with ASD and that it did so in the same way and to the same extent as in a matched control group. We found no evidence to support the hypothesis that deficits in exogenous spatial attention underlie the emergence of core ASD symptomatology.
Brouwer GJ & Heeger DJ. categorical clustering of the neural representation of color, Journal of Neuroscience, 33:15454-15465, 2013.
Abstract: Cortical activity was measured with functional magnetic resonance imaging (fMRI) while human subjects viewed 12 stimulus colors and performed either a color-naming or diverted attention task. A forward model was used to extract lower dimensional neural color spaces from the high-dimensional fMRI responses. The neural color spaces in two visual areas, human ventral V4 (V4v) and VO1, exhibited clustering (greater similarity between activity patterns evoked by stimulus colors within a perceptual category, compared to between- category colors) for the color-naming task, but not for the diverted attention task. Response amplitudes and signal-to-noise ratios were higher in most visual cortical areas for color naming compared to diverted attention. But only in V4v and VO1 did the cortical represen- tation of color change to a categorical color space. A model is presented that induces such a categorical representation by changing the response gains of subpopulations of color-selective neurons.
Freeman J, Ziemba CM, Heeger DJ, Simoncelli EP, Movshon JA, A functional and perceptual signature of the second visual area in primates, Nature Neuroscience, 16:974-981, 2013.
Abstract: There is no generally accepted account of the function of the second visual cortical area (V2), partly because no simple response properties robustly distinguish V2 neurons from those in primary visual cortex (V1). We constructed synthetic stimuli replicating the higher-order statistical dependencies found in natural texture images, and used them to stimulate macaque V1 and V2 neurons. Most V2 cells responded more vigorously to these textures than to control stimuli lacking naturalistic structure; V1 cells did not. fMRI measurements in humans revealed differences between V1 and V2 that paralleled the neuronal measurements. The ability of human observers to detect naturalistic structure in different types of texture was well predicted by the strength of neuronal and fMRI responses in V2 but not in V1. Together, these results reveal a novel and particular role for V2 in the representation of natural image structure.
Merriam EP, Gardner JL, Movshon JA, Heeger DJ, Modulation of visual responses by gaze direction in human visual cortex, Journal of Neuroscience, 33:9879-9889, 2013.
Abstract: To locate visual objects, the brain combines information about retinal location and direction of gaze. Studies in monkeys have demonstrated that eye position modulates the gain of visual signals with "gain fields", so that single neurons represent both retinotopic location and eye position. We wished to know whether eye position and retinotopic stimulus location are both represented in human visual cortex. Using functional magnetic resonance imaging, we measured cortical responses to stimuli that varied periodically in retinal locus, separately for each of several different gaze positions. Visually-evoked responses were periodic, following the periodic retinotopic stimulation. Only the response amplitudes depended on eye position; response phases were indistinguishable across eye positions. We used multi-voxel pattern analysis to decode eye position from the spatial pattern of response amplitudes. The decoder reliably discriminated eye position in five of the early visual cortical areas by taking advantage of a spatially heterogeneous eye-position dependent modulation of cortical activity. We conclude that responses in retinotopically-organized visual cortical areas are modulated by gain fields qualitatively similar to those previously observed neurophysiologically.
Yuval-Greenberg S, Heeger DJ, Continuous flash suppression modulates cortical activity in early visual cortex, Journal of Neuroscience, 33:9635-9643, 2013.
Abstract: A salient visual stimulus can be rendered invisible by presenting it to one eye while flashing a mask to the other eye. This procedure, called continuous flash suppression (CFS), has been proposed as an ideal way of studying awareness as it can make a stimulus imperceptible for extended periods of time. Previous studies have reported robust suppression of cortical activity in higher visual areas during CFS, but the role of primary visual cortex (V1) is still controversial. In this study, we resolve this controversy on the role of V1 in CFS and also begin characterizing the computational processes underlying CFS. Early visual cortical activity was measured with functional magnetic resonance imaging while human subjects viewed stimuli composed of target and mask, presented to the same or different eyes. Functional MRI responses in early visual cortex were smaller when target and mask were in different eyes compared to the same eye, not only for the lowest contrast target rendered invisible by CFS, but also for higher contrast targets which were visible even when presented to the eye opposite the mask. We infer that CFS is based on modulating the gain of neural responses, akin to reducing target contrast.
Grubb MA, Behrmann M, Egan R, Minshew NJ, Carrasco M, Heeger DJ, Endogenous spatial attention: evidence for intact functional in adults with autism, Autism Research, 6:108-118, 2013.
Abstract: Rapid manipulation of the attention field (i.e., the location and spread of visual spatial attention) is a critical aspect of human cognition, and previous research on spatial attention in individuals with autism spectrum disorders (ASD) has produced inconsistent results. In a series of three psychophysical experiments, we evaluated claims in the literature that individuals with ASD exhibit a deficit in voluntarily controlling the deployment and size of the spatial attention field. We measured the spatial distribution of performance accuracies and reaction times to quantify the sizes and locations of the attention field, with and without spatial uncertainty (i.e., the lack of predictability concerning the spatial position of the upcoming stimulus). We found that individuals with autism exhibited slower reactions times overall with spatial uncertainty, but the effects of attention on performance accuracies and reaction times were indistinguishable between individuals with autism and typically developing individuals, in all three experiments. These results provide evidence of intact endogenous spatial attention function in ASD.
Said CP, Heeger DJ, A model of binocular rivalry and cross-orientation suppression, PLOS Computational Biology, 9:e102991, 2013.
Abstract: Binocular rivalry and cross-orientation suppression are well-studied forms of competition in visual cortex, but models of these two types of competition are in tension with one another. Binocular rivalry occurs during the presentation of dichoptic grating stimuli, where two orthogonal gratings presented separately to the two eyes evoke strong alternations in perceptual dominance. Cross-orientation suppression occurs during the presentation of plaid stimuli, where the responses to a component grating presented to both eyes is weakened by the presence of a superimposed orthogonal grating. Conventional models of rivalry that rely on strong competition between orientation-selective neurons incorrectly predict rivalry between the components of plaids. Lowering the inhibitory weights in such models reduces rivalry for plaids, but also reduces it for dichoptic gratings. Using an exhaustive grid search, we show that this problem cannot be solved simply by adjusting the parameters of the model. Instead, we propose a robust class of models that rely on ocular opponency neurons, previously proposed as a mechanism for efficient stereo coding, to yield rivalry only for dichoptic gratings, not for plaids. This class of models reconciles models of binocular rivalry with the divisive normalization framework that has been used to explain cross-orientation. Our model makes novel predictions that we confirmed with psychophysical tests.
Said, C, Egan RD, Minshew NJ, Behrmann M, Heeger DJ, Normal binocular rivalry in autism: Implications for the excitation/inhibition hypothesis, Vision Research, 77:59-66, 2013.
Abstract: Autism is characterized by disruption in multiple dimensions of perception, emotion, language and cognition. Many hypotheses for the underlying neurophysiological basis have been proposed. Among these is the excitation/inhibition (E/I) imbalance hypothesis, which states that levels of cortical excitation and inhibition are disrupted in autism. We tested this theory in the visual system, because vision is one of the better understood systems in neuroscience, and because the E/I imbalance theory has been proposed to explain hypersensitivity to sensory stimuli in autism. We conducted two experiments on binocular rivalry, a well-studied psychophysical phenomenon that depends critically on excitation and inhibition levels in cortex. Using a computational model, we made specific predictions about how imbalances in excitation and inhibition levels would affect perception of two aspects of binocular rivalry: mixed perception (Experiment 1) and traveling waves (Experiment 2). We found no significant differences in either of these phenomena between high-functioning adults with autism and controls, and no evidence for a relationship between these measurements and the severity of autism. These results do not conclusively rule out an excitation/inhibition imbalance in visual system of those with autism, but they suggest that such an imbalance, if it exists, is likely to be small in magnitude.
Donner TH, Sagi D, Bonneh Y, Heeger DJ. Retinotopic patterns of correlated fluctuations in visual cortex reflect the dynamics of spontaneous perceptual suppression. Journal of Neuroscience, 33:2188-2198, 2013.
Abstract: While viewing certain stimuli, perception changes spontaneously in the face of constant input. For example, during "motion-induced blindness" (MIB), a small salient target spontaneously disappears and reappears when surrounded by a moving mask. Models of such bistable perceptual phenomena posit spontaneous fluctuations in neuronal activity throughout multiple stages of the visual cortical hierarchy. We used fMRI to link correlated activity fluctuations across human visual cortical areas V1 through V4 to the dynamics (rate and duration) of MIB target disappearance. We computed the correlations between the time series of fMRI activity in multiple retinotopic subregions corresponding to MIB target and mask. Linear decomposition of the matrix of temporal correlations revealed spatial patterns of activity fluctuations, regardless of whether or not these were time-locked to behavioral reports of target disappearance. The spatial pattern that dominated the activity fluctuations during MIB was spatially nonspecific, shared by all subregions, but did not reflect the dynamics of perception. By contrast, the fluctuations associated with the rate of MIB disappearance were retinotopically specific for the target subregion in V4, and the fluctuations associated with the duration of MIB disappearance states were target-specific in V1. Target- specific fluctuations in V1 have not previously been identified by averaging activity time-locked to behavioral reports of MIB disappear- ance. Our results suggest that different levels of the visual cortical hierarchy shape the dynamics of perception via distinct mechanisms, which are evident in distinct spatial patterns of spontaneous cortical activity fluctuations.
Honey CJ, Thesen T, Donner TH, Silbert SJ, Carlson CE, Devinsky O, Doyle WK, Rubin N, Heeger DJ, Hasson U, Slow cortical dynamics and the accumulation of information over long time scales, Neuron, 76:423-434, 2012.
Abstract: Making sense of the world requires us to process information over multiple timescales. We sought to identify brain regions that accumulate information over short and long timescales and to characterize the distinguishing features of their dynamics. We recorded electrocorticographic (ECoG) signals from individuals watching intact and scrambled movies. Within sensory regions, fluctuations of high-frequency (64–200 Hz) power reliably tracked instantaneous low-level properties of the intact and scrambled movies. Within higher order regions, the power fluctuations were more reliable for the intact movie than the scrambled movie, indicating that these regions accumulate information over relatively long time periods (several seconds or longer). Slow (<0.1 Hz) fluctuations of high-frequency power with time courses locked to the movies were observed throughout the cortex. Slow fluctuations were rela- tively larger in regions that accumulated information over longer time periods, suggesting a connection between slow neuronal population dynamics and temporally extended information processing.
Dinstein I, Heeger DJ, Lorenzi L, Minshew NJ, Malach R, Behrmann M, Unreliable evoked responses in autism, Neuron, 75:981-991, 2012.
Abstract: Autism has been described as a disorder of general neural processing, but the particular processing characteristics that might be abnormal in autism have mostly remained obscure. Here, we present evidence of one such characteristic: poor evoked response reliability. We compared cortical response amplitude and reliability (consistency across trials) in visual, auditory, and somatosensory cortices of high-functioning individuals with autism and con- trols. Mean response amplitudes were statistically indistinguishable across groups, yet trial-by-trial response reliability was significantly weaker in autism, yielding smaller signal-to-noise ratios in all sensory systems. Response reliability differences were evident only in evoked cortical responses and not in ongoing resting-state activity. These findings reveal that abnormally unreliable cortical responses, even to elementary nonsocial sensory stimuli, may represent a fundamental physiological alteration of neural processing in autism. The results motivate a critical expansion of autism research to determine whether (and how) basic neural processing proper- ties such as reliability, plasticity, and adaptation/ habituation are altered in autism.
Herrmann K, Heeger DJ, Carrasco M, Feature-based attention enhances performance by increasing response gain, Vision Research, 74:10-20, 2012.
Abstract: Covert spatial attention can increase contrast sensitivity either by changes in contrast gain or by changes in response gain, depending on the size of the attention field and the size of the stimulus (Herrmann, Montaser-Kouhsari, Carrasco, & Heeger, 2010), as predicted by the normalization model of attention (Reynolds & Heeger, 2009). For feature-based attention, unlike spatial attention, the model predicts only changes in response gain, regardless of whether the featural extent of the attention field is small or large. To test this prediction, we measured the contrast dependence of feature-based attention. Observers performed an orientation-discrimination task on a spatial array of grating patches. The spatial locations of the gratings were varied randomly so that observers could not attend to specific locations. Feature-based attention was manipulated with a 75%valid and 25%invalid pre-cue, and the featural extent of the attention field was manipulated by introducing uncertainty about the upcoming grating orientation. Performance accuracy was better for valid than for invalid pre-cues, consistent with a change in response gain, when the featural extent of the attention field was small (low uncertainty) or when it was large (high uncertainty) relative to the featural extent of the stimulus. These results for feature-based attention clearly differ from results of analogous experiments with spatial attention, yet both support key predictions of the normalization model of attention.
Wang HX, Heeger DJ, Landy MS, Responses to second-order texture modulations undergo surround suppression, Vision Research, 60:192-200, 2012.
Abstract: First-order (contrast) surround suppression has been well characterized both psychophysically and phys- iologically, but relatively little is known as to whether the perception of second-order visual stimuli exhibits analogous center–surround interactions. Second-order surround suppression was characterized by requiring subjects to detect second-order modulation in stimuli presented alone or embedded in a sur- round. Both contrast- (CM) and orientation-modulated (OM) stimuli were used. For most subjects and both OM and CM stimuli, second-order surrounds caused thresholds to be higher, indicative of second- order suppression. For CM stimuli, suppression was orientation-specific, i.e., higher thresholds for parallel than for orthogonal surrounds. However, the evidence for orientation specificity of suppression for OM stimuli was weaker. These results suggest that normalization, leading to surround suppression, operates at multiple stages in cortical processing.
Wang, HX, Freeman F, Merriam EP, Hasson U, Heeger DJ, Temporal eye movement strategies during naturalistic viewing, Journal of Vision, 12(1):16, 1-27, 2012
Abstract: The deployment of eye movements to complex spatiotemporal stimuli likely involves a variety of cognitive factors. However, eye movements to movies are surprisingly reliable both within and across observers. We exploited and manipulated that reliability to characterize observers’ temporal viewing strategies. Introducing cuts and scrambling the temporal order of the resulting clips systematically changed eye movement reliability. We developed a computational model that exhibited this behavior and provided an excellent fit to the measured eye movement reliability. The model assumed that observers searched for, found, and tracked a point of interest and that this process reset when there was a cut. The model did not require that eye movements depend on temporal context in any other way, and it managed to describe eye movements consistently across different observers and two movie sequences. Thus, we found no evidence for the integration of information over long time scales (greater than a second). The results are consistent with the idea that observers employ a simple tracking strategy even while viewing complex, engaging naturalistic stimuli.
Josipovic Z, Dinstein I, Weber J, Heeger DJ, Influence of meditation on anti-correlated networks in the brain, Frontiers in Human Neuroscience, 5:183, 1-11, 2012.
Abstract: Human experience can be broadly divided into those that are external and related to interaction with the environment, and experiences that are internal and self-related. The cerebral cortex appears to be divided into two corresponding systems: an "extrinsic" system composed of brain areas that respond more to external stimuli and tasks and an "intrinsic" system composed of brain areas that respond less to external stimuli and tasks. These two broad brain systems seem to compete with each other, such that their activity levels over time is usually anti-correlated, even when subjects are "at rest" and not performing any task. This study used meditation as an experimental manipulation to test whether this competition (anti-correlation) can be modulated by cognitive strategy. Participants either fixated without meditation (fixation), or engaged in nondual awareness (NDA) or focused attention (FA) meditations. We computed inter-area correlations ("functional connectivity") between pairs of brain regions within each system, and between the entire extrinsic and intrinsic systems. Anti-correlation between extrinsic vs. intrinsic systems was stronger during FA meditation and weaker during NDA meditation in comparison to fixation (without mediation). However, correlation between areas within each system did not change across conditions. These results suggest that the anti-correlation found between extrinsic and intrinsic systems is not an immutable property of brain organization and that practicing different forms of meditation can modulate this gross functional organization in profoundly different ways.
Carandini M, Heeger DJ, Normalization as a canonical neural computation, Nature Reviews Neuroscience, 13:51-62, 2012.
Abstract: There is increasing evidence that the brain relies on a set of canonical neural computations, repeating them across brain regions and modalities to apply similar operations to different problems. A promising candidate for such a computation is normalization, in which the responses of neurons are divided by a common factor that typically includes the summed activity of a pool of neurons. Normalization was developed to explain responses in the primary visual cortex and is now thought to operate throughout the visual system, and in many other sensory modalities and brain regions. Normalization may underlie operations such as the representation of odours, the modulatory effects of visual attention, the encoding of value and the integration of multisensory information. Its presence in such a diversity of neural systems in multiple species, from invertebrates to mammals, suggests that it serves as a canonical neural computation.
Brouwer G, Heeger DJ, Cross-orientation suppression in human visual cortex, Journal of Neurophysiology, 106:2108-2119, 2011.
Abstract: Cross-orientation suppression was measured in human primary visual cortex (V1) to test the normalization model. Subjects viewed vertical target gratings (of varying contrasts) with or without a superimposed horizontal mask grating (fixed contrast). We used functional magnetic resonance imaging (fMRI) to measure the activity in each of several hypothetical channels (corresponding to subpopulations of neurons) with different orientation tunings and fit these orientation-selective responses with the normalization model. For the V1 channel maximally tuned to the target orientation, responses increased with target contrast but were suppressed when the horizontal mask was added, evident as a shift in the contrast gain of this channel’s responses. For the channel maximally tuned to the mask orientation, a constant baseline response was evoked for all target contrasts when the mask was absent; responses decreased with increasing target contrast when the mask was present. The normalization model provided a good fit to the contrast-response functions with and without the mask. In a control experiment, the target and mask presentations were temporally interleaved, and we found no shift in contrast gain, i.e., no evidence for suppression. We conclude that the normalization model can explain cross-orientation suppression in human visual cortex. The approach adopted here can be applied broadly to infer, simultaneously, the responses of several subpopulations of neurons in the human brain that span particular stimulus or feature spaces, and characterize their interactions. In addition, it allows us to investigate how stimuli are represented by the inferred activity of entire neural populations.
Pestilli F, Carrasco M, Heeger DJ, Gardner, JL, Attentional enhancement via selection and pooling of early sensory responses in human visual cortex, Neuron, 72:832-846, 2011.
Abstract: The computational processes by which attention improves behavioral performance were characterized by measuring visual cortical activity with functional magnetic resonance imaging as humans performed a contrast-discrimination task with focal and distributed attention. Focal attention yielded robust improvements in behavioral performance accompanied by increases in cortical responses. Quantitative analysis revealed that if performance were limited only by the sensitivity of the measured sensory signals, the improvements in behavioral performance would have corresponded to an unrealistically large reduction in response variability. Instead, behavioral performance was well characterized by a pooling and selection process for which the largest sensory responses, those most strongly modulated by attention, dominated the perceptual decision. This characterization predicts that high contrast distracters that evoke large responses should negatively impact behavioral performance. We tested and confirmed this prediction. We conclude that attention enhanced behavioral performance predominantly by enabling efficient selection of the behaviorally relevant sensory signals.
Freeman J, Donner TH Heeger DJ, Inter-area correlations in the ventral visual pathway reflect feature integration, Journal of Vision, 11(4):15, 1-23, 2011.
Abstract: During object perception, the brain integrates simple features into representations of complex objects. A perceptual phenomenon known as visual crowding selectively interferes with this process. Here, we use crowding to characterize a neural correlate of feature integration. Cortical activity was measured with functional magnetic resonance imaging, simultaneously in multiple areas of the ventral visual pathway (V1-V4 and the visual word form area, VWFA, which responds preferentially to familiar letters), while human subjects viewed crowded and uncrowded letters. Temporal correlations between cortical areas were lower for crowded letters than for uncrowded letters, especially between V1 and VWFA. These differences in correlation were retinotopically specific, persisted when attention was diverted from the letters, and they disappeared when we substituted the letters with grating patches that were not crowded under our stimulus conditions. We conclude that inter-area correlations reflect feature integration, and are disrupted by crowding. We propose that crowding may perturb the transformations between neural representations along the ventral pathway that underlie the integration of features into objects.
Hallum LE, Landy MS, Heeger DJ, Human primary visual cortex (V1) is selective for second-order spatial frequency, Journal of Neurophysiology, 105:2121-2131, 2011.
Abstract: A variety of cues can differentiate objects from their surrounds. These include "first-order" cues such as luminance modulations and "second-order" cues involving modulations of orientation and contrast. Human sensitivity to first-order modulations is well described by a computational model involving spatially localized filters that are selective for orientation- and spatial frequency (SF). It is widely held that first-order modulations are represented by the firing rates of simple and complex cells ("first-order" neurons) in primary visual cortex (V1) that, likewise, have spatially localized receptive fields that are selective for orientation- and SF. Human sensitivity to second-order modulations is well described by a filter-rectify-filter (FRF) model, with first- and second-order filters selective for orientation and SF. However, little is known about how neuronal activity in visual cortex represents second-order modulations. We tested the FRF model by using an fMRI-adaptation protocol to characterize the selectivity of activity in visual cortex to second-order, orientation-defined gratings of two different SFs. fMRI responses throughout early visual cortex exhibited selective adaptation to these stimuli. The low-SF grating was a more effective adapter than the high-SF grating, incompatible with the FRF model. To explain the results, we extended the FRF model by incorporating normalization, yielding a filter-rectify-normalize-filter (FRNF) model, in which normalization enhances selectivity for second-order SF, but only for low spatial frequencies. We conclude that neurons in human visual cortex are selective for 2nd-order SF, that normalization (surround suppression) contributes to this selectivity, and that the selectivity in higher visual areas is simply fed forward from V1.
Freeman J, Brouwer GJ, Heeger DJ, Merriam EP, Orientation decoding depends on maps not columns, Journal of Neuroscience, 31:4792-4804, 2011.
Abstract: The representation of orientation in primary visual cortex (V1) has been examined at a fine spatial scale corresponding to the columnar architecture. We present functional magnetic resonance imaging (fMRI) measurements providing evidence for a topographic map of orientation preference in human V1 at a much coarser scale, in register with the angular-position component of the retinotopic map of V1. This coarse-scale orientation map provides a parsimonious explanation for why multivariate pattern analysis methods succeed in decoding stimulus orientation from fMRI measurements, challenging the widely-held assumption that decoding results reflect sampling of spatial irregularities in the fine-scale columnar architecture. Decoding stimulus attributes and cognitive states from fMRI measurements has proven useful for a number of applications, but our results demonstrate that the interpretation cannot assume decoding reflects or exploits columnar organization.
Bonneh YS, Donner TH, Sagi D, Fried, M, Cooperman A, Heeger DJ, Arieli A, Motion-induced blindness and microsaccades: cause and effect, Journal of Vision, 10(14):22, 1-15, 2010.
Abstract: It has been suggested that subjective disappearance of visual stimuli results from a spontaneous reduction of microsaccade rate causing image stabilization, enhanced adaptation and a consequent fading. In motion-induced-blindness (MIB) salient visual targets disappear intermittently when surrounded by a moving pattern. We investigated whether changes in microsaccade rate can account for MIB. We first determined that the moving mask does not affect microsaccade metrics (rate, magnitude, and temporal distribution).We then compared the dynamics of microsaccades during reported illusory disappearance (MIB) and physical disappearance (Replay) of a salient peripheral target. We found large modulations of microsaccade rate following perceptual transitions, whether illusory (MIB) or real (Replay). For MIB, the rate also decreased prior to disappearance and increased prior to reappearance.Importantly, MIB persisted in the presence of microsaccades although sustained microsaccade rate was lower during invisible than visible periods. These results suggest that the microsaccade system reacts to changes in visibility, but microsaccades also modulate MIB. The latter modulation is well described by a Poisson model of the perceptual transitions assuming that the probability for reappearance and disappearance is modulated following a microsaccade. Our results show that microsaccades counteract disappearance, but are neither necessary nor sufficient to account for MIB.
Herrmann K, Montaser-Kouhsari L, Carrasco M, Heeger DJ, When size matters: attention affects performance by contrast or response gain, Nature Neuroscience, 13:1554-1559, 2010.
Abstract: Covert attention, the selective processing of visual information in the absence of eye movements, improves behavioral performance. Here, we show that attention, both exogenous (involuntary) and endogenous (voluntary), can affect performance by contrast or response gain changes, depending on the stimulus size and the relative size of the attention field. These two variables were manipulated in a cueing task while varying stimulus contrast. We observed a change in behavioral performance consonant with a change in contrast gain for small stimuli paired with spatial uncertainty, but a change in response gain for large stimuli presented at one location (no uncertainty) and surrounded by irrelevant flanking distracters. A complementary neuroimaging experiment revealed that observers attention field was wider with than without spatial uncertainty. Our results support key predictions of the normalization model of attention, and reconcile previous, seemingly contradictory, findings on the effects of visual attention.
Larsson J, Heeger DJ, Landy, MS, Orientation selectivity of motion-boundary responses in human visual cortex, Journal of Neurophysiology, 104:2940-2950, 2010.
Abstract: Motion boundaries (local changes in visual motion direction) arise naturally when objects move relative to an observer. In human visual cortex, neuroimaging studies have identified a region (the kinetic occipital area, KO) that responds more strongly to motion-boundary stimuli than to transparent-motion stimuli. Recent fMRI studies suggest that KO may encompass multiple visual areas, and single-unit studies in macaque visual cortex have identified neurons selective for motion-boundary orientation in areas V2, V3, and V4, implying that motion-boundary selectivity may not be restricted to a single area. It is not known whether fMRI responses to motion boundaries are selective for motion-boundary orientation, as would be expected if these responses reflected the population activity of motion-boundary-selective neurons. We used an event-related fMRI adaptation protocol to measure orientation-selective responses to motion boundaries in human visual cortex. On each trial, we measured the response to a probe stimulus presented after an adapter stimulus (a vertical or horizontal motion-boundary grating). The probe stimulus was either a motion-boundary grating oriented parallel or orthogonal to the adapter stimulus, or a transparent-motion stimulus. Orientation-selective adaptation for motion boundaries - smaller responses for trials in which test and adapter stimuli were parallel to each other - was observed in multiple extrastriate visual areas. The strongest adaptation, relative to the unadapted responses, was found in V3A, V3B, LO1, LO2, and V7. Most of the visual areas that exhibited orientation-selective adaptation in our data also showed response preference for motion boundaries over transparent motion, indicating that most of the human visual areas previously shown to respond to motion boundaries are also selective for motion-boundary orientation. These results suggest that neurons selective for motion-boundary orientation are distributed across multiple human visual cortical areas and argue against the existence of a single region or area specialized for motion-boundary processing.
Offen S, Gardner J, Schluppeck D, Heeger DJ, Differential roles for the frontal eye fields (FEFs) and intraparietal sulcus (IPS) in visual working memory and visual attention, Journal of Vision, 10(11):28, 1-14, 2010.
Abstract: Cortical activity was measured with functional magnetic resonance imaging to probe the involvement of the superior precentral sulcus (including putative human frontal eye elds, FEFs) and the intraparietal sulcus (IPS) in visual short-term memory and visual attention. In two experimental tasks, human subjects viewed two visual stimuli separated by a variable delay period. The tasks placed differential demands on short-term memory and attention, but the stimuli were visually identical until after the delay period. An earlier study (S. Offen, D. Schluppeck, & D. J. Heeger, Vision Research, 2009) had found a dissociation in early visual cortex that suggested different computational mechanisms underlying the two processes. In contrast, the results reported here show that the patterns of activation in prefrontal and parietal cortex were different from one another but were similar for the two tasks. In particular, the FEF showed evidence for sustained delay period activity for both the working memory and the attention task, while the IPS did not show evidence for sustained delay period activity for either task. The results imply differential roles for the FEF and IPS in these tasks; the results also suggest that feedback of sustained activity from frontal cortex to visual cortex might be gated by task demands.
Kang MS, Lee SH, Kim J, Heeger DJ, Blake R, Modulation of spatiotemporal dynamics of binocular rivalry by collinear facilitation and pattern-dependent adaptation, Journal of Vision, 10(11):3, 1-15, 2010.
Abstract: The role of collinear facilitation was investigated to test predictions of a model for traveling waves of dominance during binocular rivalry (H. Wilson, R. Blake, & S. Lee, 2001). In Experiment 1, we characterized traveling wave dynamics using a recently developed technique called periodic perturbation (M.-S. Kang, D. Heeger, & R. Blake, 2009). Results reveal that the propagation speed of waves for a collinear stimulus increased regardless of whether that stimulus was suppressed (replicating earlier work) or dominant; this latter finding is contrary to the model's prediction. In Experiment 2, we measured perceptual dominance durations within a localized region in the center of two rival stimuli that varied in degree of collinearity. Results reveal that increased collinearity did not change average dominance durations regardless of the rivalry phase of the stimulus, again contrary to the model's prediction. Incorporating pattern-dependent modulation of adaptation rate into the model accounted for results from both experiments. Using model simulations, we show how interactions between collinear facilitation and pattern-dependent adaptation may influence the dynamics of binocular rivalry. We also discuss alternative interpretations of our findings, including the possible role of surround suppression.
Brennan J, Nir Y, Hasson U, Malach R, Heeger DJ, Pylkkanen L, Syntactic structure building in the anterior temporal lobe during natural story listening, Brain and Language, 120:163-173, 2012.
Abstract: The neural basis of syntax is a matter of substantial debate. In particular, the inferior frontal gyrus (IFG), or Broca's area, has been prominently linked to syntactic processing, but the anterior temporal lobe has been reported to be activated instead of IFG when manipulating the presence of syntactic structure. These findings are difficult to reconcile because they rely on different laboratory tasks which tap into distinct computations, and may only indirectly relate to natural sentence processing. Here we assessed neural correlates of syntactic structure building in natural language comprehension, free from artificial task demands. Subjects passively listened to Alice in Wonderland during functional magnetic resonance imaging and we correlated brain activity with a word-by-word measure of the amount syntactic structure analyzed. Syntactic structure building correlated with activity in the left anterior temporal lobe, but there was no evidence for a correlation between syntactic structure building and activity in inferior frontal areas. Our results suggest that the anterior temporal lobe computes syntactic structure under natural conditions.
Dinstein I, Thomas C, Humphreys K, Minshew N, Behrmann M, Heeger DJ. Normal movement selectivity in autism, Neuron, 66:461-469, 2010.
Abstract: It has been proposed that individuals with autism have difculties understanding the goals and intentions of others because of a fundamental dysfunction in the mirror neuron system. Here, however, we show that individuals with autism exhibited not only normal fMRI responses in mirror system areas during observation and execution of hand movements but also exhibited typical movement-selective adaptation (repetition suppression) when observing or executing the same movement repeatedly. Movement selectivity is a dening characteristic of neurons involved in movement perception, including mirror neurons, and, as such, these ndings argue against a mirror system dysfunction in autism.
Hasson U, Malach R, Heeger DJ. Reliability of cortical activity during natural stimulation, Trends in Cognitive Sciences, 14:40-48, 2010.
Abstract: Response reliability is complementary to more conventional measurements of response amplitudes, and can reveal phenomena that response amplitudes do not. Here we review studies that measured reliability of cortical activity within or between human subjects in response to naturalistic stimulation (e.g. free viewing of movies). Despite the seemingly uncontrolled nature of the task, some of these complex stimuli evoke highly reliable, selective and time-locked activity in many brain areas, including some regions that show little response modulation in most conventional experimental protocols. This activity provides an opportunity to address novel questions concerning natural vision, temporal scale of processing, memory and the neural basis of inter-group differences.
Lauritzen TZ, D'Esposito M, Heeger DJ, Silver MA. Top-down flow of visual spatial attention signals from parietal to occipital cortex, Journal of Vision, 9(13):18, 1-14, 2009.
Abstract: Given the complexity of our visual environment, the ability to selectively attend to certain locations, while ignoring others, is crucial for reducing the amount of visual information to manageable levels and for optimizing behavioral performance. Sustained allocation of spatial attention causes persistent increases in functional magnetic resonance imaging (fMRI) signals in portions of early visual cortex that retinotopically represent the attended location, even in the absence of a visual stimulus. Here we test the hypothesis that topographically organized posterior parietal cortical areas IPS1 and IPS2 transmit top-down spatial attention signals to early visual cortex. We employed fMRI and coherency analysis to measure functional connectivity among cortical areas V1, V2, V3, V3A, V3B, V7, IPS1, and IPS2 during sustained visual spatial attention. Attention increased the magnitude of coherency for many pairs of areas in occipital and parietal cortex. Additionally, attention-related activity in IPS1 and IPS2 led activity in several visual cortical areas by a few hundred milliseconds. These results are consistent with transmission of top-down spatial attention signals from IPS1 and IPS2 to early visual cortex.
Brouwer GJ & Heeger DJ. Decoding and reconstructing color from responses in human visual cortex, Journal of Neuroscience, 29:13992-14003, 2009.
Abstract: How is color represented by spatially distributed patterns of activity in visual cortex? Functional magnetic resonance imaging responses to several stimulus colors were analyzed with multivariate techniques: conventional pattern classification, a forward model of idealized color tuning, and principal component analysis (PCA). Stimulus color was accurately decoded from activity in V1, V2, V3, V4, and VO1 but not LO1, LO2, V3A/B, or MT+. The conventional classifier and forward model yielded similar accuracies, but the forward model (unlike the classifier) also reliably reconstructed novel stimulus colors not used to train (specify parameters of) the model. The mean responses, averaged across voxels in each visual area, were not reliably distinguishable for the different stimulus colors. Hence, each stimulus color was associated with a unique spatially distributed pattern of activity, presumably reflecting the color selectivity of cortical neurons. Using PCA, a color space was derived from the covariation, across voxels, in the responses to different colors. In V4 and VO1, the first two principal component scores (main source of variation) of the responses revealed a progression through perceptual color space, with perceptually similar colors evoking the most similar responses. This was not the case for any of the other visual cortical areas, including V1, although decoding was most accurate in V1. This dissociation implies a transformation from the color representation in V1 to reflect perceptual color space in V4 and VO1.
Moradi F & Heeger DJ. Inter-ocular contrast normalization in human visual cortex. Journal of Vision, 9(3):13, 1-22, 2009.
Abstract: The brain combines visual information from the two eyes and forms a coherent percept, even when inputs to the eyes are different. However, it is not clear how inputs from the two eyes are combined in visual cortex. We measured fMRI responses to single gratings presented monocularly, or pairs of gratings presented monocularly or dichoptically with several combinations of contrasts. Gratings had either the same orientation or orthogonal orientations (i.e., plaids). Observers performed a demanding task at xation to minimize top-down modulation of the stimulus-evoked responses. Dichoptic presentation of compatible gratings (same orientation) evoked greater activity than monocular presentation of a single grating only when contrast was low (<10%). A model that assumes linear summation of activity from each eye failed to explain binocular responses at 10% contrast or higher. However, a model with binocular contrast normalization, such that activity from each eye reduced the gain for the other eye, tted the results very well. Dichoptic presentation of orthogonal gratings evoked greater activity than monocular presentation of a single grating for all contrasts. However, activity evoked by dichoptic plaids was equal to that evoked by monocular plaids. Introducing an onset asynchrony (stimulating one eye 500 ms before the other, which under attentive vision results in ash suppression) had no impact on the results; the responses to dichoptic and monocular plaids were again equal. We conclude that when attention is diverted, inter-ocular suppression in V1 can be explained by a normalization model in which the mutual suppression between orthogonal orientations does not depend on the eye of origin,nor on the onset times, and cross-orientation suppression is weaker than inter-ocular (same orientation) suppression.
Kang M-S, Heeger DJ, Blake R. Periodic perturbations producing phase-locked fluctuations in visual perception. Journal of Vision, 9(2):8 1-22, 2009.
Abstract: This paper describes a novel psychophysical and analytical technique, called periodic perturbation, for creating and characterizing perceptual waves associated with transitions in visibility of a stimulus during binocular rivalry and during binocular fusion. Observers tracked rivalry within a small, central region of spatially extended rival targets while small, brief increments in contrast (triggers) were presented repetitively in antiphase within different regions of the two rival targets. Appropriately timed triggers produced entrainment of rivalry alternations within the central region, with the optimal timing dependent on an observer s native alternation rate. The latency between trigger and state switch increased with the distance between the location of the trigger and the central region being monitored, providing evidence for traveling waves of dominance. Traveling waves produced by periodic perturbation exhibited the same characteristics as those generated using a less efcient, more demanding discrete trial technique. We used periodic perturbation to reveal a novel relation between the dynamics associated with the spontaneous perceptual alternations and the speed of traveling waves across observers. In addition, we found evidence for traveling waves even when the events triggering them were initiated within regions of the visual eld where binocular vision was stable, in the absence of binocular rivalry, implying that perceptual organization generally depends on spatio-temporal context.
Reynolds JH & Heeger DJ. The normalization model of attention. Neuron, 61:168-185, 2009.
Abstract: Attention has been found to have a wide variety of effects on the responses of neurons in visual cortex. We describe a model of attention that exhibits each of these different forms of attentional modulation, depending on the stimulus conditions and the spread (or selectivity) of the attention field in the model. The model helps reconcile proposals that have been taken to represent alternative theories of attention. We argue that the variety and complexity of the results reported in the literature emerge from the variety of empirical protocols that were used, such that the results observed in any one experiment depended on the stimulus conditions and the subject's attentional strategy, a notion that we define precisely in terms of the attention field in the model, but that has not typically been completely under experimental control.
Offen S, Schluppeck D, Heeger DJ. The role of early visual cortex in visual short-term memory and visual attention. Vision Research, 49:1352-1362, 2009.
Abstract: We measured cortical activity with functional magnetic resonance imaging to probe the involvement of early visual cortex in visual short-term memory and visual attention. In four experimental tasks, human subjects viewed two visual stimuli separated by a variable delay period. The tasks placed differential demands on short-term memory and attention, but the stimuli were visually identical until after the delay period. Early visual cortex exhibited sustained responses throughout the delay when subjects performed attention-demanding tasks, but delay-period activity was not distinguishable from zero when subjects performed a task that required short-term memory. This dissociation reveals different computational mechanisms underlying the two processes.
Dinstein I, Gardner JL, Jazayeri M, Heeger DJ. Executed and observed movements have different distributed representations in human aIPS. Journal of Neuroscience, 28:11231-11239, 2008.
Abstract: How similar are the representations of executed and observed hand movements in the human brain? We used functional magnetic resonance imaging (fMRI) and multivariate pattern classification analysis to compare spatial distributions of cortical activity in response to several observed and executed movements. Subjects played the rockpaperscissors game against a videotaped opponent, freely choosing their movement on each trial and observing the opponents hand movement after a short delay. The identities of executed movements were correctly classified from fMRI responses in several areas of motor cortex, observed movements were classified from responses in visual cortex, and both observed and executed movements were classified from responses in either left or right anterior intraparietal sulcus (aIPS). We interpret above chance classification as evidence for reproducible, distributed patterns of cortical activity that were unique for execution and/or observation of each movement. Responses in aIPS enabled accurate classification of movement identity within each modality (visual or motor), but did not enable accurate classification across modalities (i.e., decoding observed movements from a classifier trained on executed movements and vice versa). These results support theories regarding the central role of aIPS in the perception and execution of movements. However, the spatial pattern of activity for a particular observed movement was distinctly different from that for the same movement when executed, suggesting that observed and executed movements are mostly represented by distinctly different subpopulations of neurons in aIPS.
Donner TH, Sagi D, Bonneh Y, Heeger DJ. Opposite neural signatures of motion-induced blindness in human dorsal and ventral visual cortex. Journal of Neuroscience, 28:10298-10310, 2008.
Abstract: Motion-induced blindness (MIB) is a visual phenomenon in which a salient static target spontaneously fluctuates in and out of visual awareness when surrounded by a moving mask pattern. It has been hypothesized that MIB reflects an antagonistic interplay between cortical representations of the static target and moving mask. Here, we report evidence for such antagonism between human ventral and dorsal visual cortex during MIB. Functional magnetic resonance imaging (fMRI) responses in ventral visual area V4 decreased with the subjective disappearance of the target. These response decreases were specific for the cortical subregion corresponding retinotopically to the target, occurred early in time with respect to the perceptual report, and could not be explained by shifts of attention in reaction to target disappearance. At the same time, responses increased in mask-specific subregions in dorsal visual areas in and around the intraparietal sulcus. These opposite responses in ventral and dorsal visual areas occurred only during subjective target disappearance, not when the target was physically removed. Perceptual reports of target disappearance were furthermore associated with a global modulation of activity, which was delayed in time, and evident throughout early visual cortex, for both subjective target disappearance and physical target removal. We conclude that awareness of the target is tightly linked to the strength of its representation in ventral visual cortex, and that the mask representation in dorsal visual cortex plays a crucial role in the spontaneous suppression of the target representation during MIB.
Hasson U, Landesman O, Knappmeyer B, Vallines I, Rubin N, Heeger DJ. Neurocinematics: The neuroscience of films. Projections: The Journal for Movies and Mind, 2:1-26, 2008.
Abstract: While the recognition that films can impose a tight grip on viewers' minds dates back to the early days of cinema, until recently there was no way to record the mental states of viewers while watching a film. In this paper, we describe a new method for assessing the effect of a given film on viewers' brain activity. Brain activity was measured using functional magnetic resonance imaging (fMRI) during free viewing of films, and inter-subject correlation analysis (ISC, Figure 1) was used to assess similarities in the spatiotemporal responses across viewers' brains during movie watching. Our results demonstrate that some films can exert considerable control over brain activity (Figure 2) and eye movements (Figure 3). However, this was not the case for all types of motion picture sequences (Figure 4), and the level of control over viewers' brain activity differed as a function of movie content (Figure 5), editing (Figure 6), and directing style (Figure 7). We propose that ISC may be useful to film studies by providing a quantitative neuroscientific assessment (Figures 8 and 9) of the impact of different styles of filmmaking upon viewers' brains, and a valuable method for the film industry to better assess its products. Finally, we suggest that this method brings together two separate, largely unrelated disciplines, cognitive neuroscience and film studies, and may open the way for a new interdisciplinary field of "neurocinematic" studies.
Nir Y, Dinstein I, Malach R, Heeger DJ. BOLD and spiking activity - a comment on Viswanathan and Freeman. Nature Neuroscience, 11:523, 2008.
Summary: Viswanathan and Freeman (Nat Neurosci, 2007) claim that oxygen concentration and by inference blood oxygen level-dependent (BOLD) fMRI reflect synaptic activity more than spiking activity. This is a fundamental and controversial issue in fMRI research, so this claim, if incorrect, may erroneously bias the interpretation of a large body of data. The authors simultaneously recorded multi-unit activity (MUA), local field potentials (LFP), and tissue oxygen concentration in primary visual cortex of anesthetized cats stimulated with moving gratings. During high temporal frequency stimulation, when thalamic inputs are active but few cortical neurons respond, oxygen signals were observed without MUA. Hence, they concluded that oxygen responses reflect synaptic inputs more than spiking. However, careful inspection of their results leads to the opposite conclusion and supports a tight coupling between oxygen signals and local cortical spiking.
Gardner JL, Merriam EP, Movshon JA, Heeger DJ. Maps of visual space in human occipital cortex are retinotopic, not spatiotopic. Journal of Neuroscience, 28:3988-3999, 2008.
Abstract: We experience the visual world as phenomenally invariant to eye position, but almost all cortical maps of visual space in monkeys use a retinotopic reference frame - that is, the cortical representation of a point in the visual world is different across eye positions. It was recently reported that human cortical area MT (unlike monkey MT) represents stimuli in a reference frame linked to the position of stimuli in space, a spatiotopic reference frame. We used visuotopic mapping with BOLD fMRI signals to define 12 human visual cortical areas, and then determined whether the reference frame in each area was spatiotopic or retinotopic. We found that all 12 areas, including MT, represented stimuli in a retinotopic reference frame. Although there were patches of cortex in and around these visual areas that were ostensibly spatiotopic, none of these patches exhibited reliable stimulus-evoked responses. We conclude that the early, visuotopically-organized visual cortical areas in the human brain (like their counterparts in the monkey brain) represent stimuli in a retinotopic reference frame.
Hasson U, Yang E, Vallines I, Heeger DJ, Rubin N. A hierarchy of temporal receptive windows in human cortex. Journal of Neuroscience, 28:2539-2550, 2008.
Abstract: Real-world events unfold at different time scales and, therefore, cognitive and neuronal processes must likewise occur at different time scales. We present a novel procedure that identifies brain regions responsive to sensory information accumulated over different time scales. We measured functional magnetic resonance imaging activity while observers viewed silent films presented forward, backward, or piecewise-scrambled in time. Early visual areas (e.g., primary visual cortex and MT complex responsive to visual motion) exhibited high response reliability regardless of disruptions in temporal structure. In contrast, the reliability of responses in several higher brain areas, including the superior temporal sulcus (STS), precuneus, posterior lateral sulcus (LS), temporal parietal junction (TPJ), and frontal eye field (FEF), was affected by information accumulated over longer time scales. These regions showed highly reproducible responses for repeated forward, but not for backward or piecewise-scrambled presentations. Moreover, these regions exhibited marked differences in temporal characteristics, with LS, TPJ, and FEF responses depending on information accumulated over longer durations ( 36 s) than STS and precuneus ( 12 s). We conclude that, similar to the known cortical hierarchy of spatial receptive fields, there is a hierarchy of progressively longer temporal receptive windows in the human brain.
Dinstein I, Thomas C, Behrmann M, Heeger DJ. A mirror up to nature. Current Biology, 18:R13-18, 2008.
Abstract: Mirror neurons were first documented in the macaque monkey a little over ten years ago. Their discovery has led to the formulation of several theories about their function in humans, including suggestions that mirror neurons are involved in understanding the meaning and intentions of observed actions, learning by imitation, feeling empathy, formation of a "theory of mind", and even the development of language. Hypotheses have also been made about the consequences of mirror neuron dysfunction; foremost among these is the notion that such a dysfunction during development leads to many of the social and cognitive symptoms associated with the autism spectrum disorders (ASDs). Yet, despite a decade of prolific research on these appealing theories, there is little evidence to support them. In this essay, we review the current state of "mirror system" research, point to several weaknesses in the field, and offer suggestions for how better to study these remarkably interesting neurons in both neurotypical and autistic individuals.
Lee SH, Blake R, Heeger DJ. Hierarchy of responses underlying binocular rivalry. Nature Neuroscience, 10:1048-1054, 2007.
Abstract: During binocular rivalry, physical stimulation is dissociated from conscious visual awareness. Human brain imaging reveals a tight linkage between neural events in human primary visual cortex (V1) and dynamics of perceptual waves during transitions in dominance during binocular rivalry. Here, we report results from experiments in which observers’ attention was diverted from the rival stimuli, implying that: 1) competition between two rival stimuli involves neural circuits in V1, 2) attention is crucial for the consequences of this neural competition to advance to higher visual areas and promote perceptual waves.
Dinstein I, Hasson U, Rubin N, Heeger DJ. Brain areas selective for both observed and executed movements. Journal of Neurophysiology, 98:1415-1427, 2007.
Abstract: When observing a particular movement a subset of movement-selective visual and visuomotor neurons are active in the observer's brain forming a representation of the observed movement. Similarly, when executing a movement a subset of movement-selective motor and visuomotor neurons are active forming a representation of the executed movement. In this study we used an fMRI-adaptation protocol to assess cortical response selectivity to observed and executed movements simultaneously. Subjects freely played the rock-paper-scissors game against a videotaped opponent, sometimes repeatedly observing or executing the same movement on subsequent trials.Numerous brain areas exhibited adaptation (repetition suppression) during either repeated observations or repeated executions of the same movement. A subset of areasexhibited an overlap of both effects, containing neurons with selective responses for both executed and observed movements. We describe the function of these unique movement representation areas in the context of the human mirror system, which is expected to respond selectively to both observed and executed movements.
Yang Z, Heeger DJ, Seidemann E. Rapid and precise retinotopic mapping of visual cortex obtained by voltage sensitive dye imaging in the behaving monkey. Journal of Neurophysiology, 98:1002-1014, 2007.
Abstract: Retinotopy is a fundamental organizing principle of the visual cortex. Over the years, a variety of techniques have been used to examine it. None of these techniques, however, provides a way to characterize retinotopy rapidly, at the sub-millimeter range, in alert, behaving subjects. Voltage sensitive dye imaging (VSDI) can be used to monitor neuronal population activity at high spatial and temporal resolutions. Here we present a VSDI protocol for rapid and precise retinotopic mapping in the behaving monkey. Two monkeys performed a fixation task while thin visual stimuli swept periodically at a high speed in one of two possible directions through a small region of visual space. Because visual space is represented systematically across the cortical surface, each moving stimulus produced a traveling wave of activity in the cortex that could be precisely measured with VSDI. The time at which the peak of the traveling wave reached each location in the cortex linked this location with its retinotopic representation. We obtained detailed retinotopic maps from a region of ~1 cm2 over the dorsal portion of areas V1 and V2. Retinotopy obtained during less than 4 minutes of imaging had a spatial precision of 0.11-0.19 mm, was consistent across experiments, and reliably predicted the locations of the response to small localized stimuli. The ability to rapidly obtain precise retinotopic maps in behaving monkeys opens the door for detailed analysis of the relationship between spatiotemporal dynamics of population responses in the visual cortex and perceptually guided behavior.
Levy I, Schluppeck D, Heeger DJ, & Glimcher PW, Specificity of human cortical areas for reaches and saccades. Journal of Neuroscience, 27:4687-4696, 2007.
Abstract: Electrophysiological studies in monkeys have identified effector-related regions in the posterior parietal cortex (PPC). The lateral intraparietal area (LIP), for example, responds preferentially for saccades whereas the parietal reach region (PRR) responds preferentially for arm movements. However, the degree of effector selectivity actually observed is limited; each area contains neurons selective for the non-preferred effector, and many neurons in both areas respond for both effectors. We used fMRI to assess the degree of effector preference at the population level, focusing on topographically organized regions in the human PPC (V7, IPS1 and IPS2). An event-related design adapted from monkey experiments was employed. In each trial, an effector cue preceded the appearance of a spatial target, after which a go-signal instructed subjects to produce the specified movement with the specified effector. Our results show that the degree of effector specificity is limited in many cortical areas, and transitions gradually from saccade to reach preference as one moves through the hierarchy of areas in the occipital, parietal, and frontal cortices. Saccade preference was observed in visual cortex, including early areas and V7. IPS1 exhibited balanced activation to saccades and reaches, whereas IPS2 showed a weak but significant preference for reaches. In frontal cortex, areas near the central sulcus showed a clear and absolute preference for reaches while the Frontal Eye Field (FEF) showed little or no effector selectivity. Although these results contradict many theoretical conclusions about effector specificity, they are compatible with the complex picture arising from electrophysiological studies and also with previous imaging studies that reported largely overlapping saccade and arm related activation. The results are also compatible with theories of efficient coding in cortex.
Montaser-Kouhsari L, Landy MS, Heeger DJ, & Larsson J. Orientation-selective adaptation to illusory contours in human visual cortex. Journal of Neuroscience, 27:2186-2195, 2007.
Abstract: Humans can perceive illusory or subjective contours in the absence of any real physical boundaries. We used an adaptation protocol to look for orientation-selective neural responses to illusory contours defined by phase-shifted abutting line gratings in the human visual cortex. We measured fMRI responses to illusory-contour test stimuli after adapting to an illusory-contour adapter stimulus that was oriented parallel or orthogonal to the test stimulus. We found orientation-selective adaptation to illusory contours in early (V1 and V2) and higher-tier visual areas (V3, hV4, VO1, V3A/B, V7, LO1, LO2). That is, fMRI responses were smaller for test stimuli parallel to the adapter than for test stimuli orthogonal to the adapter. In two control experiments using spatially jittered and phase-randomized stimuli, we demonstrated that this adaptation was not just in response to differences in the distribution of spectral power in the stimuli. Orientation-selective adaptation to illusory contours increased from early to higher-tier visual areas. Thus, both early and higher-tier visual areas contain neurons selective for the orientation of this type of illusory contour.
Olman CA, Inati S, & Heeger DJ. The effect of large veins on spatial localization with GE BOLD at 3T: displacement, not blurring. NeuroImage, 34:1126-1135, 2007.
Abstract: We used two different methods of region of interest (ROI) definition to investigate the spatial accuracy of functional magnetic resonance imaging (fMRI) at low and high spatial resolution. The “single-condition localizer” consisted of block alternation between a target stimulus and a mean gray background. The “differential localizer” consisted of block alternation between the target stimulus and another stimulus that filled the complement of the visual field. A separate series of scans, in which the target stimulus was presented briefly with long inter-stimulus intervals, was used to measure the hemodynamic impulse response function (HIRF). As expected, the differential localizer defined more restricted ROIs that better matched the predicted cortical representation of the target stimulus. However, at low resolution (3mm isotropic) many voxels that responded positively to the target stimulus in the differential protocol responded negatively to the target stimulus in the single-condition localizer and in the HIRF measurements. The localization errors were attributed to voxels near large veins, which were identified based on low mean intensity and high variance. At high resolution (1.2mm isotropic), the effects of large veins were present, but affected a smaller number of voxels. Thus, the use of differential localizers does not necessarily result in a more accurate indication of the underlying neural activity. Localization errors are reduced at higher spatial resolutions and can be eliminated by identification and removal of voxels dominated by large veins.
Silver MA, Ress D, & Heeger DJ. Neural correlates of sustained spatial attention in human early visual cortex. J Neurophysiol, 97:229-237, 2007.
Abstract: Attention is thought to enhance perceptual performance at attended locations due to top-down attention signals that modulate activity in visual cortex. Here, we show that activity in early visual cortex is sustained during maintenance of attention in the absence of visual stimulation. We used functional magnetic resonance imaging (fMRI) to measure activity in visual cortex while human subjects performed a visual detection task in which a variable-duration delay period preceded target presentation. Portions of cortical areas V1, V2, and V3 representing the attended part of the visual field exhibited sustained increases in activity throughout the delay period. Portions of these cortical areas representing peripheral, unattended parts of the visual field displayed sustained decreases in activity. The data were well-fit by a model that assumed the sustained neural activity was constant in amplitude over a time period equal to that of the actual delay period for each trial. These results demonstrate that sustained attention responses are present in early visual cortex (including primary visual cortex) in the absence of a visual stimulus, and that these responses correlate with the allocation of visuospatial attention in both the spatial and temporal domains.
Larsson J & Heeger DJ. Two retinotopic visual areas in human lateral occipital cortex. Journal of Neuroscience, 26:13128-13142, 2006.
Abstract: We describe two visual field maps, LO1 and LO2, in human lateral occipital cortex between dorsal V3 and V5/MT+. Each map contained a topographic representation of the contralateral visual hemifield. The eccentricity representations were shared with V1/V2/V3. The polar angle representation in LO1 extended from the lower vertical meridian (at the boundary with dorsal V3) through the horizontal to the upper vertical meridian (at the boundary with LO2). The polar angle representation in LO2 was the mirror-reversal of that in LO1. LO1 and LO2 overlapped with the posterior part of the object-selective lateral occipital complex and the kinetic occipital region (KO). The retinotopy and functional properties of LO1 and LO2 suggest that they correspond to two new human visual areas, which lack exact homologues in macaque visual cortex. The topography, stimulus selectivity and anatomical location of LO1 and LO2 indicate that they integrate shape information from multiple visual submodalities in retinotopic coordinates.
Liu T, Heeger DJ, Carrasco M. Neural correlates of the visual vertical meridian asymmetry. Journal of Vision, 6:1294-1306, 2006.
Abstract: Human visual performance is better below than above fixation along the vertical meridian &## Matlab code for running the simulations (25Kb tarfile). # Matlabe code for analyzing the simulation results (25Kb tarfile).150; vertical meridian asymmetry (VMA). Here we used fMRI to investigate the neural correlates of the VMA. We presented stimuli of two possible sizes and spatial frequencies on the horizontal and vertical meridians and analyzed the fMRI data in subregions of early visual cortex (V1/V2) that corresponded retinotopically to the stimulus locations. Asymmetries in both the spatial extent and amplitude of the fMRI measurements correlated with the behavioral VMA. These results demonstrate that the VMA has a neural basis at the earliest stages of cortical visual processing, andsensory responses of large populations of neurons in visual cortex.
Schluppeck D, Curtis CE, Glimcher PW, & Heeger DJ. Sustained activity in topographic areas of human posterior parietal cortex during memory-guided saccades. J Neurosci, 26:5098-5108, 2006.
Abstract: In a previous study, we identified three cortical areas in human posterior parietal cortex that exhibited topographic responses during memory-guided saccades [visual area 7 (V7), intraparietal sulcus 1 (IPS1), and IPS2], which are candidate homologs of macaque parietal areas such as the lateral intraparietal area and parietal reach region. Here, we show that these areas exhibit sustained delay-period activity, a critical physiological signature of areas in macaque parietal cortex. By varying delay duration, we disambiguated delay-period activity from sensory and motor responses. Mean time courses in the parietal areas were well fit by a linear model comprising three components representing responses to (1) the visual target, (2) the delay period, and (3) the eye movement interval. We estimated the contributions of each component: the response amplitude during the delay period was substantially smaller (the transient visual target. All three parietal regions showed comparable delay-period response amplitudes, with a trend toward larger responses from V7 to IPS1 and IPS2. Responses to the cue and during the delay period showed clear lateralization with larger responses to trials in which the target was placed in the contralateral visual field, suggesting that both of these components contributed to the topography we measured.
Larsson J, Landy MS, & Heeger DJ. Orientation-selective adaptation to first- and second-order patterns in human visual cortex. J Neurophysiol, 95:962-881, 2006.
Abstract: Second-order textures – patterns that cannot be detected by mechanisms sensitive only to luminance changes – are ubiquitous in visual scenes, but the neuronal mechanisms mediating perception of such stimuli are not well understood. We used an adaptation protocol to measure neural activity in the human brain selective for the orientation of second-order textures. FMRI responses were measured in three subjects to presentations of first- and second-order probe gratings after adapting to a high-contrast first- or second-order grating that was either parallel or orthogonal to the probe gratings. First-order (LM) stimuli were generated by modulating the stimulus luminance. Second-order stimuli were generated by modulating the contrast (CM) or orientation (OM) of a first-order carrier. We used four combinations of adapter and probe stimuli: LM:LM, CM:CM, OM:OM, and LM:OM. The fourth condition tested for cross-modal adaptation with first-order adapter and second-order probe stimuli. Attention was diverted from the stimulus by a demanding task at fixation. Both first- and second-order stimuli elicited orientation-selective adaptation in multiple cortical visual areas, including V1, V2, V3, V3A/B, a newly identified visual area anterior to dorsal V3 which we have termed LO1, hV4, and VO1. For first-order stimuli (condition LM:LM), the adaptation was no larger in extrastriate areas than in V1, implying that the orientation-selective first-order (luminance) adaptation originated in V1. For second-order stimuli (conditions CM:CM and OM:OM), the magnitude of adaptation, relative to the absolute response magnitude, was significantly larger in VO1 (and for condition CM:CM, also in V3A/B and LO1) than in V1, suggesting that second-order stimulus orientation was extracted by additional processing after V1. There was little difference in the amplitude of adaptation between the second-order conditions. No consistent effect of adaptation was found in the cross-modal condition LM:OM, in agreement with psychophysical evidence for weak interactions between first- and second-order stimuli and computational models of separate mechanisms for first- and second-order visual processing.
Silver MA, Ress D, & Heeger DJ, Topographic maps of visual spatial attention in human parietal cortex, J Neurophysiol 94(2):1358-71, 2005.
Abstract: Functional magnetic resonance imaging (fMRI) was used to measure activity in human parietal cortex during performance of a visual detection task in which the focus of attention systematically traversed the visual field. Critically, the stimuli were identical on all trials (except for slight contrast changes in a fully randomized selection of the target locations) whereas only the cued location varied. Traveling waves of activity were observed in posterior parietal cortex consistent with shifts in covert attention in the absence of eye movements. The temporal phase of the fMRI signal in each voxel indicated the corresponding visual field location. Visualization of the distribution of temporal phases on a flattened representation of parietal cortex revealed at least two distinct topographically organized cortical areas within the intraparietal sulcus (IPS), each representing the contralateral visual field. Two cortical areas were proposed based on this topographic organization, which we refer to as IPS1 and IPS2 to indicate their locations within the IPS. This nomenclature is neutral with respect to possible homologies with well-established cortical areas in the monkey brain. The two proposed cortical areas exhibited relatively little response to passive visual stimulation in comparison with early visual areas. These results provide evidence for multiple topographic maps in human parietal cortex.
Schluppeck D, Glimcher P, & Heeger DJ, Topographic organization for delayed saccades in human posterior parietal cortex, J Neurophysiol 94(2):1372-84, 2005.
Abstract: Posterior parietal cortex (PPC) is thought to play a critical role in decision making, sensory attention, motor intention, and/or working memory. Research on the PPC in non-human primates has focused on the lateral intraparietal area (LIP) in the intraparietal sulcus (IPS). Neurons in LIP respond after the onset of visual targets, just before saccades to those targets, and during the delay period in between. To study the function of posterior parietal cortex in humans, it will be crucial to have a routine and reliable method for localizing specific parietal areas in individual subjects. Here, we show that human PPC contains at least two topographically organized regions, which are candidates for the human homologue of LIP. We mapped the topographic organization of human PPC for delayed (memory guided) saccades using fMRI. Subjects were instructed to fixate centrally while a peripheral target was briefly presented. After a further 3-s delay, subjects made a saccade to the remembered target location followed by a saccade back to fixation and a 1-s inter-trial interval. Targets appeared at successive locations "around the clock" (same eccentricity, approximately 30 degrees angular steps), to produce a traveling wave of activity in areas that are topographically organized. PPC exhibited topographic organization for delayed saccades. We defined two areas in each hemisphere that contained topographic maps of the contra-lateral visual field. These two areas were immediately rostral to V7 as defined by standard retinotopic mapping. The two areas were separated from each other and from V7 by reversals in visual field orientation. However, we leave open the possibility that these two areas will be further subdivided in future studies. Our results demonstrate that topographic maps tile the cortex continuously from V1 well into PPC.
Lee SH, Blake R, & Heeger DJ, Traveling waves of activity in primary visual cortex during binocular rivalry, 8:22-23, 2005.
Abstract: When the two eyes view large dissimilar patterns that induce binocular rivalry, alternating waves of visibility are experienced, as one pattern sweeps the other out of conscious awareness. Here we show tight linkage between dynamics of perceptual waves during rivalry and neural events in human primary visual cortex (V1).
Neri P, Bridge H & Heeger DJ, Stereoscopic processing of absolute and relative disparity in human visual cortex, Journal of Neurophysiology, 92:1880-1891, 2004.
Abstract: Stereoscopic vision relies mainly on relative depth differences between objects, rather than on their absolute distance in depth from where the eyes fixate. However, relative disparities are computed from absolute disparities, and it is not known where these two stages are represented in the human brain. Using functional magnetic resonance imaging (fMRI), we assessed absolute and relative disparity selectivity with stereoscopic stimuli consisting of pairs of transparent planes in depth in which the absolute and relative disparity signals could be independently manipulated (at a local spatial scale). In experiment 1, relative disparity was kept constant, while absolute disparity was varied in half the blocks of trials (“mixed” blocks) and kept constant in the remaining half (“same” blocks), alternating between blocks. Because neuronal responses undergo adaptation and reduce their firing rate following repeated presentation of an effective stimulus, the fMRI signal reflecting activity of units selective for absolute disparity is expected to be smaller during “same” blocks as compared to “mixed” ones. Experiment 2 similarly manipulated relative disparity rather than absolute disparity. The results from both experiments were consistent with adaptation with differential effects across visual areas such that 1) dorsal areas (V3a, MT+/V5, V7) showed more adaptation to absolute than to relative disparity; 2) ventral areas (hV4, V8/V4) showed an equal adaptation to both; 3) early visual areas (V1, V2, V3) showed a small effect in both experiments. These results indicate that processing in dorsal areas may rely mostly on information about absolute disparities, while ventral areas split neural resources between the two types of stereoscopic information so as to maintain an important representation of relative disparity.
Zenger-Landolt B & Heeger DJ, Response suppression in V1 agrees with psychophysics of surround masking , Journal of Neuroscience, 23:6884-6893, 2003.
Abstract: When a target stimulus is embedded in a high contrast surround, the target appears reduced in contrast and is harder to detect, and neural responses in visual cortex are suppressed. We used functional magnetic resonance imaging (fMRI) and psychophysics to quantitatively compare these physiological and perceptual effects. Observers performed a contrast discrimination task on a contrast-reversing sinusoidal target grating. The target was either presented in isolation or embedded in a high-contrast surround. While observers performed the task, we also measured fMRI responses as a function of target contrast, both with and without a surround.Wefound that the surround substantially increased the psychophysical thresholds while reducing fMRI responses. The two data sets were compared, on the basis of the assumption that a fixed response difference is required for correct discrimination, and we found that the psychophysics accounted for 96.5% of the variance in the measured V1 responses. The suppression in visual areas V2 and V3 was stronger, too strong to agree with psychophysics. The good quantitative agreement between psychophysical thresholds and V1 responses suggests V1 as a plausible candidate for mediating surround masking.
Ress D & Heeger DJ, Neuronal correlates of perception in early visual cortex, Nature Neuroscience, 6:414-420, 2003.
Abstract: We used functional magnetic resonance imaging (fMRI) to measure activity in human early visual cortex (areas V1, V2 and V3) during a challenging contrast-detection task. Subjects attempted to detect the presence of slight contrast increments added to two kinds of background patterns. Behavioral responses were recorded so that the corresponding cortical activity could be grouped into the usual signal detection categories: hits, false alarms, misses and correct rejects. For both kinds of background patterns, the measured cortical activity was retinotopically specific. Hits and false alarms were associated with significantly more cortical activity than were correct rejects and misses. That false alarms evoked more activity than misses indicates that activity in early visual cortex corresponded to the subjects' percepts, rather than to the physically presented stimulus.
Carandini M, Heeger DJ, & Senn W, A synaptic explanation of suppression in visual cortex, Journal of Neuroscience, 22:10053–10065, 2002.
Abstract: The responses of neurons in the primary visual cortex (V1) are suppressed by mask stimuli that do not elicit responses if presented alone. This suppression is widely believed to be mediated by intr\ acortical inhibition. As an alternative, we propose that it can be explained by thalamocortical synaptic depression. This explanation correctly predicts that suppression is monocular, immune to cortical adap\ tation, and occurs for mask stimuli that elicit responses in the thalamus but not in the cortex. Depression also explains other phenomena previously ascribed to intracortical inhibition. It explains why resp\ onses saturate at high stimulus contrast, whereas selectivity for orientation and spatial frequency is invariant with contrast. It explains why transient responses to flashed stimuli are nonlinear, whereas s\ patial summation is primarily linear. These results suggest that the very first synapses into the cortex, and not the cortical network, may account for important response properties of V1 neurons.
Huk A, Dougherty RF, & Heeger DJ, Retinotopy and functional subdivision of human areas MT and MST, Journal of Neuroscience, 22:7195-7205, 2002.
Abstract: We performed a series of functional magnetic resonance imaging experiments to divide the human MT+ complex into subregions that may be identified as homologs to a pair of macaque motion-responsive visual areas: the middle temporal area (MT) and the medial superior temporal area (MST). Using stimuli designed to tease apart differences in retinotopic organization and receptive field size, we established a double dissociation between two distinct MT+ subregions in 8 of the 10 hemispheres studied. The first subregion exhibited retinotopic organization but did not respond to peripheral ipsilateral stimulation, indicative of smaller receptive fields. Conversely, the second subregion within MT+ did not demonstrate retinotopic organization but did respond to peripheral stimuli in both the ipsilateral and contralateral visual hemifields, indicative of larger receptive fields. We tentatively identify these subregions as the human homologues of macaque MT and MST, respectively. Putative human MT and MST were typically located on the posterior/ventral and anterior/dorsal banks of a dorsal/posterior limb of the inferior temporal sulcus, similar to their relative positions in the macaque superior temporal sulcus.
Neri P & Heeger DJ, Spatiotemporal mechanisms for detecting and identifying image features in human vision, Nature Neuroscience, 5:812-816, 2002.
Abstract: The human visual system constantly selects salient features in the environment for further attention, processing and identification. Models of feature detection often assume that salient features are selected on the basis of contrast energy (local variance in intensity in the visual stimulus. This hypothesis, however, has not been tested directly. We used psychophysical reverse correlation to study how humans detect and identify basic image features (bars and short line segments). Subjects detected a briefly-flashed 'target bar' that was embedded in 'noise bars' that randomly changed in intensity over space and time. By studying how the intensity of the noise bars affected performance, we were able to dissociate two processing stages: an early 'detection' stage, whereby only locations of high contrast energy in the image were selected, and an identification stage (~100 ms later) during which subjects used image intensity at selected locations to determine whether the target was bright or dark.
Heeger DJ & Ress D, What does fMRI tell us about neuronal activity?, Nature Reviews Neuroscience, 3:142-151, 2002.
Abstract: In recent years, cognitive neuroscientists have taken great advantage of functional magnetic resonance imaging (fMRI) as a non-invasive method of measuring neuronal activity in the human brain. But what exactly does fMRI tell us? We know that its signals arise from changes in local haemodynamics that, in turn, result from alterations in neuronal activity, but exactly how neuronal activity, haemodynamics and fMRI signals are related is unclear. It has been assumed that the fMRI signal is proportional to the local average neuronal activity, but many factors can influence the relationship between the two. A clearer understanding of how neuronal activity influences the fMRI signal is needed if we are correctly to interpret functional imaging data.
Huk AC & Heeger DJ, Pattern-motion responses in human visual cortex, Nature Neuroscience, 5:72-75, 2001.
Abstract: Physiological models of visual motion processing posit that 'pattern-motion cells' represent the direction of moving objects independent of their particular spatial pattern. We performed fMRI experiments to identify pattern-motion cells in the human brain, and to test the hypothesis that the activity of these neurons is linked to the perception of coherent motion. A protocol employing moving 'plaid' stimuli allowed us to separate pattern-motion responses from other types of motion-related activity within the same brain structures, and revealed strong pattern-motion selectivity in human visual area MT+. Reducing the perceptual coherence of the plaids yielded a corresponding decrease in pattern-motion responsivity, providing evidence that percepts of coherent motion are closely linked to the activity of pattern-motion cells.
Huk AC, Ress D, & Heeger DJ, Neuronal basis of the motion aftereffect reconsidered. Neuron, 32:161–172, 2001.
Abstract: Several recent fMRI studies have reported response increases in human MT+ correlated with perception of the motion aftereffect (MAE). However, MT+ responses can be strongly affected by attention, and subjects may naturally attend more strongly during the MAE than during controls without MAE. We found that requiring subjects to attend to the motion of the stimulus on both MAE and control trials produced equal levels of MT+ response, suggesting that attention may be a major confound in the interpretation of previous fMRI MAE experiments; in our data, attention appears to account for the entire effect. After eliminating this confound, we sought to measure direction-selective motion adaptation in human visual cortex. We observed that adaptation produced a direction-selective imbalance in MT+ responses (as well as earlier visual areas including V1), and yielded a corresponding psychophysical asymmetry in speed discrimination thresholds. These findings provide physiological evidence of a population-level response imbalance related to the MAE, and quantify the relative proportions of direction-selective neurons in human cortical visual areas.
Backus BT, Fleet DJ, Parker AJ, & Heeger DJ, Human cortical activity correlates with stereoscopic depth perception. J Neurophysiol, 86:2054-2068 , 2001.
Abstract: Stereoscopic depth perception is based on binocular disparities. Although neurons in primary visual cortex (V1) are selective for binocular disparity, their responses do not explicitly code perceived depth. The stereoscopic pathway must therefore include additional processing beyond V1. We used functional magnetic resonance imaging (fMRI) to examine stereo processing in V1 and other areas of visual cortex. We created stereoscopic stimuli that portrayed two planes of dots in depth, placed symmetrically about the plane of fixation, or else asymmetrically with both planes either nearer or farther than fixation. The interplane disparity was varied parametrically to determine the stereoacuity threshold (the smallest detectable disparity) and the upper depth limit (largest detectable disparity). fMRI was then used to quantify cortical activity across the entire range of detectable interplane disparities. Measured cortical activity covaried with psychophysical measures of stereoscopic depth perception. Activity increased as the interplane disparity increased above the stereoacuity threshold and dropped as interplane disparity approached the upper depth limit. From the fMRI data and an assumption that V1 encodes absolute retinal disparity, we predicted that the mean response of V1 neurons should be a bimodal function of disparity. A post hoc analysis of electrophysiological recordings of single neurons in macaques revealed that, although the average firing rate was a bimodal function of disparity (as predicted), the precise shape of the function cannot fully explain the fMRI data. Although there was widespread activity within the extrastriate cortex (consistent with electrophysiological recordings of single neurons), area V3A showed remarkable sensitivity to stereoscopic stimuli, suggesting that neurons in V3A may play a special role in the stereo pathway.
Simoncelli EP & Heeger DJ, Representing retinal image speed in visual cortex. Nature Neuroscience, 4:461-462, 2001.
Abstract: Speed preferences in MT neurons are found to be unaffected by changes in stimulus pattern, supporting the hypothesis that these neurons represent retinal image velocities.
Polonsky A, Blake R, Braun J, & Heeger DJ, Neuronal activity in human primary visual cortex correlates with perception during binocular rivalry. Nature Neuroscience, 3:1153-1159, 2000.
Abstract: During binocular rivalry two incompatible monocular images compete for perceptual dominance, with one pattern temporarily suppressed from conscious awareness. We measured fMRI signals in early visual cortex while subjects viewed rival dichoptic images of two different contrasts; the contrast difference served as a “tag” for the representations of the two monocular images. Activity in primary visual cortex (V1) increased when subjects perceived the higher contrast pattern and decreased when subjects perceived the lower contrast pattern. These fluctuations in V1 activity during rivalry were about 55% as large as those evoked by alternately presenting the two monocular images without rivalry. The rivalry-related fluctuations in V1 activity were roughly equal to those observed in extrastriate visual areas (V2, V3, V3a, and V4v). These results challenge the view that binocular rivalry primarily takes place later in the cortical visual pathways.
Ress D, Backus BT, & Heeger DJ, Activity in primary visual cortex predicts performance in a visual detection task. Nature Neuroscience, 3:940-945, 2000.
Abstract: Visual attention can affect both neural activity and behavior in humans. To quantify possible links between the two, we measured activity in early visual cortex (V1, V2 and V3) during a challenging pattern detection task. Activity was dominated by a large response that was independent of the presence or absence of the stimulus pattern. The measured activity quantitatively predicted the subject's pattern detection performance: when activity was greater, the subject was more likely to correctly discern the presence or absence of the pattern. This stimulus independent activity had several characteristics of visual attention, suggesting that attentional mechanisms modulate activity in early visual cortex, and that this attention related activity strongly influences performance.
Heeger DJ, Huk AC, Geisler WS, & Albrecht DG, Spikes versus BOLD: what does neuroimaging tell us about neuronal activity? Nature Neuroscience, 3:631-633, 2000.
Abstract: By demonstrating that fMRI responses in human MT+ increase linearly with motion coherence and comparing these responses with slopes of single-neuron firing rates in monkey MT, a new paper provides the best evidence so far that fMRI responses are proportional to firing rates.
Xing J, & Heeger DJ, Measurement and modeling of center-surround suppression and enhancement. Vision Research, 41:571-583, 2001.
Abstract: The apparent contrast of a central stimulus is affected by the presence of surrounding stimuli. For some stimulus conditions, the apparent contrast is suppressed and for other conditions the apparent contrast is enhanced. This report is intended to offer a coherent description of the stimulus factors that influence suppression and enhancement. Using a contrast-matching protocol, we measured the contrast dependence of center-surround interactions by systematically varying the suprathreshold contrasts of the central and surround gratings. Different spatial configurations of the surround stimuli were studied. Our results confirmed previous findings that (1) a surround stimulus could produce either contrast enhancement or contrast suppression depending on the balance of the central and surround contrasts; (2) suppression varied with the width of the surround stimulus and was strongly orientation-specific; and (3) enhancement was less sensitive to changes in surround configurations (in particular, enhancement did not depend on the colinearity of the central and surround gratings). Based on the experimental data, we developed a computational model to account for center-surround suppression and enhancement.
Xing J, & Heeger DJ, Center-surround interactions in foveal and peripheral vision. Vision Research, 40:3065-3072, 2000.
Abstract: The perceived contrast of a central stimulus can be decreased (surround suppression) or increased (surround facilitation) by the presence of surround stimuli. In this report we examined center-surround interactions in foveal and peripheral vision using contrast-matching tasks. We found that: (1) surround suppression became markedly stronger as the center-surround stimulus was moved toward the periphery; (2) surround facilitation diminished in the periphery; and (3) the suppression in the periphery was less orientation- and frequency-specific than that in the fovea, so that significant suppression was induced even when the central and surround gratings had very different orientations and spatial frequencies. The different center-surround interactions in the fovea and periphery can not be accounted for by cortical magnification, suggesting that center-surround interactions in the fovea and periphery are incommensurable and play different functional roles in human image processing.
Huk AC & Heeger DJ, Task-related modulation of visual cortex. J Neurophysiol, 83:3525–3536, 2000.
Abstract: We performed a series of experiments to quantify the effects of task performance on cortical activity in early visual areas. Functional magnetic resonance imaging (fMRI) was used to measure cortical activity in several cortical visual areas including primary visual cortex (V1) and the MT complex (MT+) as subjects performed a variety of threshold-level visual psychophysical tasks. Performing speed, direction, and contrast discrimination tasks produced strong modulations of cortical activity. For example, one experiment tested for selective modulations of MT+ activity as subjects alternated between performing contrast and speed discrimination tasks. MT+ responses modulated in phase with the periods of time during which subjects performed the speed discrimination task; that is, MT+ activity was higher during speed discrimination than during contrast discrimination. Task related modulations were consistent across repeated measurements in each subject; however, significant individual differences were observed between subjects. Together, the results suggest 1) that specific changes in the cognitive/behavioral state of a subject can exert selective and reliable modulations of cortical activity in early visual cortex, even in V1; 2) that there are significant individual differences in these modulations; and 3) that visual areas and pathways that are highly sensitive to small changes in a given stimulus feature (such as contrast or speed) are selectively modulated during discrimination judgments on that feature. Increasing the gain of the relevant neuronal signals in this way may improve their signal-to-noise to help optimize task performance.
Nestares O & Heeger DJ, Robust multiresolution alignment of MRI brain volumes, Magnetic Resonance in Medicine, 43:705-715, 2000.
Abstract: An algorithm for the automatic alignment of MRI volumes of the human brain was developed, based on techniques adopted from the computer vision literature for image motion estimation. Most image registration techniques rely on the assumption that corresponding voxels in the two volumes have equal intensity, which is not true for MRI volumes acquired with different coils and/or pulse sequences. Intensity normalization and contrast equalization were used to minimize the differences between the intensities of the two volumes. However, these preprocessing steps do not correct perfectly for the image differences when using different coils and/or pulse sequences. Hence, the alignment algorithm relies on robust estimation, which automatically ignores voxels where the intensities are sufficiently different in the two volumes. A multiresolution pyramid implementation enables the algorithm to estimate large displacements. The resulting algorithm is used routinely to align MRI volumes acquired using different protocols (3D SPGR and 2D fast spin echo) and different coils (surface and head) to subvoxel accuracy (better than 1 mm).
Heeger DJ, Boynton GM, Demb JB, Seidemann E, & Newsome WT, Motion Opponency in Visual Cortex, Journal of Neuroscience, 19:7162-7174 1999.
Abstract: Perceptual studies suggest that visual motion perception is mediated by opponent mechanisms that correspond to mutually suppressive populations of neurons sensitive to motions in opposite directions. We tested for a neuronal correlate of motion opponency using functional magnetic resonance imaging to measure brain activity in human visual cortex. There was strong motion opponency in a secondary visual cortical area known as the human MT complex (MT+), but there was little evidence of motion opponency in primary visual cortex (V1). To determine whether the level of opponency in human MT+ and monkey MT are comparable, a variant of these experiments was performed using multi-unit electrophysiological recording in areas MT and MST of the macaque monkey brain. While there was substantial variability in the degree of opponency between recording sites, the monkey and human data were qualitatively similar on average. These results provide further evidence that: 1) direction selective signals underlie human MT+ responses, 2) neuronal signals in human MT+ support visual motion perception, 3) human MT+ may be homologous to macaque monkey MT along with adjacent motion sensitive brain areas, and 4) that fMRI measurements are correlated with average spiking activity.
Heeger DJ, Linking Visual Perception with Human Brain Activity, Current Opinion in Neurobiology, 9:474-479.
Abstract: The past year has seen great advances in the use of functional magnetic resonance imaging (fMRI) to study the functional organization of human visual cortex, to measure the neuronal correlates of visual perception, and to test computational theories of vision. This paper reviews quantitative fMRI methods and summarizes some recent results that illustrate the promise of the approach.
Gandhi SP, Heeger DJ, and Boynton GM, Spatial Attention Affects Brain Activity in Human Primary Visual Cortex, Proc Natl Acad Sci USA, 96:3314-3319 1999.
Abstract: Functional magnetic resonance imaging (fMRI) was used to test if instructing subjects to attend to one or another location in a visual scene would affect neural activity in human primary visual cortex (V1). Stimuli were moving gratings restricted to a pair of peripheral, circular apertures, positioned to the right and to the left of a central fixation point. Subjects were trained to perform a motion discrimination task, attending (without moving their eyes) at any moment in time to one of the two stimulus apertures. FMRI responses were recorded while subjects were cued to alternate their attention between the two apertures. V1 responses in each hemisphere modulated with the alternation of the cue; responses were greater when the subject attended to the stimuli in the contralateral hemifield. The attentional modulation of the brain activity was about 25 percent of that evoked by alternating the stimulus with a uniform field.
Demb JB, Boynton GM, and Heeger DJ, Functional Magnetic Resonance Imaging of Early Visual Pathways in Dyslexia, J Neurosci, 18:6939-6951, 1998.
Abstract: We measured brain activity, perceptual thresholds and reading performance in a group of dyslexic and normal readers to test the hypothesis that dyslexia is associated with an abnormality in the magnocellular (M) pathway of the early visual system. Functional magnetic resonance imaging (fMRI) was used to measure brain activity in conditions designed to preferentially stimulate the M pathway. Speed discrimination thresholds, that measure the minimal increase in stimulus speed that is just noticeable, were acquired in a paradigm modeled after a previous study of M pathway lesioned monkeys. Dyslexics showed reduced brain activity compared to controls both in primary visual cortex (V1) and in several extrastriate areas, including area MT+ that is believed to receive a predominant M pathway input. There was a strong three-way correlation between brain activity, speed discrimination thresholds, and reading speed. Subjects with higher V1 and MT+ responses had lower perceptual thresholds (better performance) and were faster readers. These results support the hypothesis for an M pathway abnormality in dyslexia and imply strong relationships between the integrity of the M pathway, visual motion perception, and reading ability.
Boynton GM, Demb JB, Glover GH, and Heeger DJ, Neural Basis of Contrast Discrimination, Vision Research, 39:257-269, 1999.
Abstract: Psychophysical contrast increment thresholds were compared with neuronal responses, measured using functional magnetic resonance imaging (fMRI), to test the hypothesis that pattern discrimination judgments are limited by neuronal signals in early visual cortical areas. FMRI was used to measure human brain activity as a function of stimulus contrast, in each of several identifiable visual cortical areas. Contrast increment thresholds were measured for the same stimuli across a range of baseline contrasts. FMRI responses in visual areas V1, V2d, and V3d were found to be consistent with the psychophysical judgments, i.e., a contrast increment was detected when the fMRI responses in each of these brain areas increased by a criterion amount. Thus, the pooled activity of large numbers of neurons can reasonably well predict behavioral performance.
Demb JB, Boynton GM, Best M, and Heeger DJ, Psychophysical evidence for a magnocellular pathway deficit in dyslexia, Vision Research, 38:1555-1560, 1998.
Abstract: The relationship between reading ability and psychophysical performance was examined to test the hypothesis that dyslexia is associated with a deficit in the magnocellular (M) pathway. Speed discrimination thresholds and contrast detection thresholds were measured under conditions (low mean luminance, low spatial frequency, high temporal frequency) for which psychophysical performance presumably depends on M pathway integrity. Dyslexic subjects had higher psychophysical thresholds than controls in both the speed discrimination and contrast detection tasks, but only the differences in speed thresholds were statistically significant. In addition, there was a strong correlation between individual differences in speed thresholds and reading rates. These results support the hypothesis for an M pathway abnormality in dyslexia, and suggest that motion discrimination may be a better indicator of dyslexia than is contrast sensitivity.
Carandini M, Heeger DJ, and Movshon JA, Linearity and Gain Control in V1 Simple Cells, in Cerebral Cortex, vol. 13: Models of Cortical Circuits, 1999.
Simoncelli EP and Heeger DJ, A Model of Neuronal Responses in Visual Area MT. Vision Research, 38:743-761, 1998.
Abstract: Electrophysiological studies indicate that neurons in the Middle Temporal (MT) area of the primate brain are selective for the velocity of visual stimuli. This paper describes a computational model of MT physiology, in which local image velocities are represented via the distribution of MT neuronal responses. The computation is performed in two stages, corresponding to neurons in cortical areas V1 and MT. Each stage computes a weighted linear sum of inputs, followed by rectification and divisive normalization. V1 receptive field weights are designed for orientation and direction selectivity. MT receptive field weights are designed for velocity (both speed and direction) selectivity. The paper includes computational simulations accounting for a wide range of physiological data, and describes experiments that could be used to further test and refine the model.
Black M, Sapiro G, Marimont D, and Heeger DJ, Robust anisotropic diffusion, IEEE Transactions on Image Processing, 7:421-432, 1998.
Abstract: Relations between anisotropic diffusion and robust statistics are described in this paper. Specifically, we show that anisotropic diffusion can be seen as a robust estimation procedure that estimates a piecewise smooth image from a noisy input image. The “edge-stopping” function in the anisotropic diffusion equation is closely related to the error norm and in uence function in the robust estimation framework. This connection leads to a new “edge-stopping” function based on Tukey’s biweight robust estimator that preserves sharper boundaries than previous formulations and improves the automatic stopping of the diffusion. The robust statistical interpretation also provides a means for detecting the boundaries (edges) between the piecewise smooth regions in an image that has been smoothed with anisotropic diffusion. Additionally, we derive a relationship between anisotropic diffusion and regularization with line processes. Adding constraints on the spatial organization of the line processes allows us to develop new anisotropic diffusion equations that result in a qualitative improvement in the continuity of edges.
Demb JB, Boynton GM, and Heeger DJ, Brain activity in visual cortex predicts individual differences in reading performance, Proc Natl Acad Sci USA, 94:13363-13366, 1997
Abstract: The relationship between brain activity and reading performance was examined to test the hypothesis that dyslexia involves a deficit in a specific visual pathway known as the magnocellular (M) pathway. Functional magnetic resonance imaging (fMRI) was used to measure brain activity in dyslexic and control subjects in conditions designed to preferentially stimulate the M pathway. Dyslexics showed reduced activity compared to controls both in primary visual cortex (V1) and in a secondary cortical visual area (MT+) that is believed to receive a strong M pathway input. Most importantly, significant correlations were found between individual differences in reading rate and brain activity. These results support the hypothesis for an M pathway abnormality in dyslexia and imply a strong relationship between the integrity of the M pathway and reading ability.
Carandini M, Heeger DJ, and Movshon JA, Linearity and normalization of simple cells of the macaque primary visual cortex, J Neurosci, 17:8621-8644, 1997.
Abstract: Simple cells in the primary visual cortex often appear to compute a weighted sum of the light intensity distribution of the visual stimuli that fall on their receptive fields. A linear model of these cells has the advantage of simplicity and captures a number of basic aspects of cell function. It, however, fails to account for important response nonlinearities, such as the decrease in response gain and latency observed at high contrasts and the effects of masking by stimuli that fail to elicit responses when presented alone. To account for these nonlinearities we have proposed a normalization model, which extends the linear model to include mutual shunting inhibition among a large number of cortical cells. Shunting inhibition is divisive, and its effect in the model is to normalize the linear responses by a measure of stimulus energy. To test this model we performed extracellular recordings of simple cells in the primary visual cortex of anesthetized macaques. We presented large stimulus sets consisting of (1) drifting gratings of various orientations and spatiotemporal frequencies; (2) plaids composed of two drifting gratings; and (3) gratings masked by full-screen spatiotemporal white noise. We derived expressions for the model predictions and fitted them to the physiological data. Our results support the normalization model, which accounts for both the linear and the nonlinear properties of the cells. An alternative model, in which the linear responses are subject to a compressive nonlinearity, did not perform nearly as well.
Tolhurst DJ and Heeger DJ, Contrast normalization and a linear model for the directional selectivity of simple cells in cat striate cortex, Visual Neuroscience, 14:19-26, 1997.
Abstract: Previous tests of the linearity of spatiotemporal summation in cat simple cells have compared the responses to moving sinusoidal gratings and to gratings whose contrast was modulated sinusoidally in time. In particular, since a moving grating can be expressed as a sum of modulated gratings, the response to a moving grating should be predictable (assuming linearity) from the responses to modulated gratings. However, these simple linear predictions have shown varying degrees of failure (e.g. Reid et al., 1987, 1991), depending on the directional selectivity of the neurons (Tolhurst & Dean, 1991). We demonstrate here that the failures of these linear predictions are, in fact, explained by the contrast normalization model of Heeger (1993). We concentrate on the ratio of the measured to predicted moving grating responses. In the context of the contrast normalization model, calculating this ratio turns out to be particularly appropriate, since the ratio is independent of the precise details of the linear fronted mechanisms ultimately responsible for directional selectivity. Hence, the contrast normalization model can be compared quantitatively with this ratio measure, by varying only one free parameter. When account is taken both of the expansive output nonlinearity and of contrast normalization, the directional selectivity of simple cells seems to be dependent only on linear spatiotemporal filtering.
Tolhurst DJ and Heeger DJ, Comparison of contrast normalization and threshold models of the responses of simple cells in cat striate cortex, Visual Neuroscience, 14:293-309, 1997.
Abstract: In almost every study of the linearity of spatiotemporal summation in simple cells of the cat's visual cortex, there have been systematic mismatches between the experimental observations and the predictions of the linear theory. These mismatches have generally been explained by supposing that the initial spatiotemporal summation stage is strictly linear, but that the following output stage of the simple cell is subject to some contrast dependent nonlinearity. Two main models of the output nonlinearity have been proposed: the threshold model (e.g. Tolhurst & Dean, 1987) and the contrast normalization model (e.g. Heeger, 1992a,b). In this paper, the two models are fitted rigorously to a variety of previously published neurophysiological data, in order to determine whether one model is a better explanation of the data. We reexamine data on the interaction between two bar stimuli presented in different parts of the receptive field; on the relationship between the receptive field map and the inverse Fourier transform of the spatial frequency tuning curve; on the dependence of response amplitude and phase on the spatial phase of stationary gratings; on the relationships between the responses to moving and modulated gratings; and on the suppressive action of gratings moving in a neuron's nonpreferred direction. In many situations, the predictions of the two models are similar, but the contrast normalization model usually fits the data slightly better than the threshold model, and it is easier to apply the equations of the normalization model. More importantly, the normalization model is naturally able to account very well for the details and subtlety of the results in experiments where the total contrast energy of the stimuli changes; some of these phenomena are completely beyond the scope of the threshold model. Rigorous application of the models' equations has revealed some situations where neither model fits quite well enough, and we must suppose, therefore, that there are some subtle nonlinearities still to be characterized.
Fleet DJ, Wagner H, and Heeger DJ, Modelling Binocular Neurons in the Primary Visual Cortex, in Computational and Biological Mechanisms of Visual Coding, M. Jenkin and L. Harris, eds, Cambridge University Press, p. 103-130, 1997.
Abstract: This chapter presents a formal description and analysis of a binocular energy model of disparity selectivity. According to this model, disparity selectivity results from a combination of position-shifts and/or phase-shifts. Our theoretical analysis suggests how one might perform an experiment to estimate the relative contributions of phase and position shifts to the disparity selectivity of binocular neurons, based on their responses to drifting sinusoidal grating stimuli of different spatial frequencies and disparities.
We also show that for drifting gratings stimuli, the binocular energy response (with phase and/or position shifts) is a sinusoidal function of disparity, consistent with the physiology of neurons in primary visual cortex (area 17) of the cat. However, Freeman and Ohzawa (1990) also found that the depth of modulation in the sinusoidal disparity tuning curves was remarkably invariant to interocular contrast differences. This is inconsistent with the binocular energy model.
As a consequence we propose a modified binocular energy model that incorporates two stages of divisive normalization. The first normalization stage is monocular, preceding the combination of signals from the two eyes. The second normalization stage is binocular. Our simulation results demonstrate that the normalized binocular energy model provides the required stability of the depth of response modulation. Simulations also demonstrate that the model's monocular and binocular contrast response curves are consistent with those of neurons in primary visual cortex.
Fleet DJ & Heeger DJ. Embedding invisible information in color images, in Proceedings of International Conference on Image Processing, 1997.
Abstract: We describe a method for embedding information in color images. A model of human color vision is used to ensure that the embedded signal is invisible. Sinusoidal signals are embedded so that they can be detected (decoded) without the use of the original image. The sinusoids act as a grid, providing a coordinate frame on the image. We use the grid to automatically scale and align (deskew) images that have been printed and then scanned.
Nestares O and Heeger DJ, Modeling the Apparent Frequency Specific Suppression in Simple Cell Responses, Vision Research, 37:1535-1543, 1997.
Abstract: Simple cells in cat striate cortex are selective for spatial frequency. It is widely believed that this selectivity arises simply because of the way in which the neur ons sum inputs from the lateral geniculate nucleus. Alternate models, however, advocate the need for frequency-specific inhibitory mechanisms to refine the spatial frequency se lectivity. Indeed, simple cell responses are often suppressed by superimposing stimuli with spatial frequencies that flank the neuron's preferred spatial frequency.
In this article, we compare two models of simple cell responses head-to-head. One of these models, the flanking-suppression model, includes an inhibitory mechanism that is spec ific to frequencies that flank the neuron's preferred spatial frequency. The other model, the nonspecific-suppression model, includes a suppressive mechanism that is very broad ly tuned for spatial frequency. Both models also include a rectification nonlinearity and both may include an additional accelerating (e.g., squaring) output nonlinearity. We d emonstrate that both models can be consistent with the apparent flanking suppression. However, based on other experimental results, we argue that the nonspecific-suppression mo del is more plausible. We conclude that the suppression is probably broadly tuned for spatial frequency and that the apparent flanking suppression is actually due to distortion s introduced by an accelerating output nonlinearity.
Tian TY, Tomasi C, and Heeger DJ, Comparison of Approaches to Egomotion Computation, Proceedings of Computer Vision and Pattern Recognition, 1996.
Abstract: We evaluated six algorithms for computing egomotion from image velocities. We established benchmarks for quantifying bias and sensitivity to noise, and for quantifying the convergence properties of those algorithms that require numerical search. Our simulation results reveal some interesting and surprising results. First, it is often written in the literature that the egomotion problem is difficult because translation (e.g., along the X-axis) and rotation (e.g., about the Y-axis) produce similar image velocities. We found, to the contrary, that the bias and sensitivity of our six algorithms are totally invariant with respect to the axis of rotation. Second, it is also believed by some that fixating helps to make the egomotion problem easier. We found, to the contrary, that fixating does not help when the noise is independent of the image velocities. Fixation does help if the noise is proportional to speed, but this is only for the trivial reason that the speeds are slower under fixation. Third, it is widely believed that increasing the field of view will yield better performance. We found, to the contrary, that this is not necessarily true.
Boynton GM, Engel SA, Glover GH, and Heeger DJ, Linear Systems Analysis of fMRI in Human V1, J Neurosci, 16:4207-4221, 1996.
Abstract: The linear transform model of functional magnetic resonance imaging (fMRI) hypothesizes that fMRI responses are proportional to local average neural activity, averaged over a period of time. This article reports results from three empirical tests that support this hypothesis. First, fMRI responses in human primary visual cortex (V1) depend separably on stimulus timing and stimulus contrast. Secondly, responses to long duration stimuli can be predicted from responses to shorter duration stimuli. Thirdly, the noise in the fMRI data is independent of stimulus contrast and temporal period. Although these tests can not prove the correctness of the linear transform model, they might have been used to reject the model. Since the linear transform model is consistent with our data, we proceeded to estimate the temporal fMRI impulse response function and the underlying (presumably neural) contrast-response function of human V1.
Fleet DJ, Wagner H, and Heeger DJ, Encoding of Binocular Disparity: Energy Models, Position Shifts and Phase Shifts, Vision Research, 36:1839-1858, 1996.
Abstract: Neurophysiological data supports two models for the disparity selectivity of binocular simple and complex cells in the primary visual cortex. These involve binocular combinations of monocular receptive fields that are shifted in retinal position (the position-shift model) or in phase (the phase-shift model) between the two eyes. This article presents a theoretical analysis of these two models. We describe the quantitative behaviour of these model neurons, along with proposals for how one might measure the relative contributions of phase- and position-shifts towards the disparity selectivity of binocular cells. The analysis also reveals ambiguities in the disparity encoding that is inherent in these model neurons, suggesting a need for a second stage of processing; we propose that pooling the binouclar responses across orientations and scales (spatial frequency) is capable of producing an unambiguous representation of disparity.
Heeger DJ, Simoncelli EP, and Movshon JA, Computational Models of Cortical Visual Processing, Proc Natl Acad Sci USA, 93:6 23-627, 1996.
Abstract: The visual responses of neurons in the cerebral cortex were first adequately characterized in the 1960s by D. H. Hubel and T. N. Wiesel [(1962) J. Physiol. (London) 160, 106-154; (1968) J. Physiol. (London) 195, 215-243] using qualitative analyses based on simple geometric visual targets. Over the past 30 years, it has become common to consider the properties of these neurons by attempting to make formal descriptions of the transformations they execute on the visual image. Most such models have their roots in linear-systems approaches pioneered in the retina by C. Enroth-Cugell and J. R. Robson [(1966) J. Physiol. (London) 187, 517-552], but it is clear that purely linear models of cortical neurons are inadequate. We present two related models: one designed to account for the responses of simple cells in primary visual cortex (V1) and one designed to account for the responses of pattern direction selective cells in MT (or V5), an extrastriate visual area thought to be involved in the analysis of visual motion. These models share a common structure that operates in the same way on different kinds of input, and instantiate the widely held view that computational strategies are similar throughout the cerebral cortex. Implementations of these models for Macintosh microcomputers are available and can be used to explore the models' properties.
Heeger AJ, Heeger DJ, Langen J, and Yang T, Image Enhancement using Polymer Grid Triode Arrays, Science, 270:1642-1644, 1995.
Abstract: An array of polymer grid triodes with common grid functions as a "plastic retina" which provides local contrast gain control for image enhancement. The polymer grid triode array can be implemented on the focal plane to process the analog data directly from a photodetector array. Alternatively, the array of polymer grid triodes can be utilized after analog to digital conversion and integrated directly into a display.
Heeger DJ and Bergen JR, Pyramid Based Texture Analysis/Synthesis, Computer Graphics Proceedings, p. 229-238, 1995.
Abstract: This paper describes a method for synthesizing images that match the texture appearance of a given digitized sample. This synthesis is completely automatic and requires only the ``target'' texture as input. It allows generation of as much texture as desired so that any object can be covered. It can be used to produce solid textures for creating textured 3-d objects without the distortions inherent in texture mapping. It can also be used to synthesize texture mixtures, images that look a bit like each of several digitized samples. The approach is based on a model of human texture perception, and has potential to be a practically useful tool for graphics applications.
Heeger DJ, The Representation of Visual Stimuli in Primary Visual Cortex, Current Directions in Psychological Science, 3:159-163, 1994.
Carandini M, and Heeger DJ, Summation and Division by Neurons in Visual Cortex. Science, 264:1333-1336, 1994.
Abstract: Recordings from monkey primary visual cortex (V1) were used to test a model for the visually-driven responses of simple cells. According to the model, simple cells compute a linear sum of the responses of lateral geniculate nucleus (LGN) neurons. In addition, each simple cell's linear response is divided by the pooled activity of a large number of other simple cells. The cell membrane performs both operations; synaptic currents are summed and then divided by the total membrane conductance. Current and conductance are decoupled (by a complementary arrangement of excitation and inhibition) so that current depends only on the LGN inputs and conductance depends only on the cortical inputs. Closed form expressions were derived for fitting and interpreting physiological data. The model accurately predicted responses to drifting grating stimuli of various contrasts, orientations, and spatiotemporal frequencies.
Teo P and Heeger DJ, Perceptual Image Distortion, Proceedings of SPIE, volume 2179, p. 127-141, 1994.
Abstract: In this paper, we present a perceptual distortion measure that predicts image integrity far better than mean-squared error. This perceptual distortion measure is based on a model of human visual processing that fits empirical measurements of: (1) the response properties of neurons in the primare visual cortex, and (2) the psychophysics of spatial pattern detection. We also illustrate the usefulness of the model in predicting perceptual distortion in real images.
Teo P and Heeger DJ, Perceptual Image Distortion, First IEEE International Conference on Image Processing, vol 2, pp 982-986, November 1994.
Abstract: In this paper, we present a perceptual distortion measure that predicts image integrity far better than mean-squared error. This perceptual distortion measure is based on a model of human visual processing that fits empirical measurements of the psychophysics of spatial pattern detection. The model of human visual processing proposed involves two major components: a steerable pyramid transform and contrast normalization. We also illustrate the usefulness of the model in predicting perceptual distortion in real images.
Heeger DJ, Modeling simple cell direction selectivity with normalized, half-squared, linear operators, Journal of Neurophysiology, 70:1885-1898, 1993.
Summary: 1. A longstanding view of simple cells is that they sum their inputs linearly. However, the linear model falls short of a complete account of simple-cell direction selectivity. We have developed a nonlinear model of simple-cell responses (hereafter referred to as the normalization model) to explain a larger body of physiological data. 2. The normalization model consists of an underlying linear stage along with two additional nonlinear stages. The first is a half-squaring nonlinearity; half-squaring is half-wave rectification followed by squaring. The second is a divisive normalization non-linearity in which each model cell is suppressed by the pooled activity of a large number of cells. 3. By comparing responses with counterphase (flickering) gratings and drifting gratings, researchers have demonstrated that there is a nonlinear contribution to simple-cell responses. Specifically they found 1) that the linear prediction from counterphase grating responses underestimates a direction index computed from drifting grating responses, 2) that the linear prediction correctly estimates responses to gratings drifting in the preferred direction, and 3) that the linear prediction overestimates responses to gratings drifting in the nonpreferred direction. 4. We have simulated model cell responses and derived mathematical expressions to demonstrate that the normalization model accounts for this empirical data. Specifically the model behaves as follows. 1) The linear prediction from counterphase data underestimates the direction index computed from drifting grating responses. 2) The linear prediction from counterphase data overestimates the response to gratings drifting in the nonpreferred direction. The discrepancy between the linear prediction and the actual response is greater when using higher contrast stimuli. 3) For an appropriate choice of contrast, the linear prediction from counterphase data correctly estimates the response to gratings drifting in the preferred direction. For higher contrasts the linear prediction overestimates the actual response, and for lower contrasts the linear prediction underestimates the actual response. 5. In addition, the normalization model is qualitatively consistent with data on the dynamics of simple-cell responses. Tolhurst et al. found that simple cells respond with an initial transient burst of activity when a stimulus first appears. The normalization model behaves similarly; it takes some time after a stimulus first appears before the model cells are fully normalized. We derived the dynamics of the model and found that the transient burst of activity in model cells depends in a particular way on stimulus contrast. The burst is short for high-contrast stimuli and longer for low-contrast stimuli.
Chichilnisky EJ, Heeger DJ, and Wandell BA, Functional segregation of color and motion perception examined in motion nulling, Vision Research, 15:2113-2125, 1993.
Abstract: We examine two hypotheses about the functional segregation of color and motion perception, using a motion nulling task. The most common interpretation of functional segregation, that motion perception depends only on one of the three dimensions of color, is rejected. We propose and test an alternative formulation of functional segregation: that motion perception depends on a univariate motion signal driven by all three color dimensions, and that the motion signal is determined by the product of the stimulus contrast and a term that depends only on the relative cone excitations. Two predictions of this model are confirmed. First, motion nulling is transitive: when two stimuli null a third they also null another. Second, motion nulling is homogeneous: if two stimuli null one another, they continue to null one another when their contrasts are scaled equally. We describe how to apply our formulation of functional segregation to other behavioral and physiological measurements.
Heeger DJ and Simoncelli EP, Model of visual motion sensing, in Spatial Vision in Humans and Robots, Harris L and Jenkin M, eds, Cambridge University Press, p. 367-392, 1993.
Abstract: A number of researchers have proposed models of early motion sensing based on direction-selective, spatiotemporal linear operators. Others have formalized the problem of measuring optical flow in terms of the spatial and temporal derivatives of stimulus intensity. Recently, the spatiotemporal filter models and the gradient-based methods have been placed into a common framework. In this chapter, we review that framework and we extend it to develop a new model for the computation and representation of velocity information in the visual system. We use the model to simulate psychophysical data on perceived velocity of sine-grating plaid patterns, and to simulate physiological data on responses of simple cells in primary (striate) visual cortex.
Jepson A and Heeger DJ, Linear subspace methods for recovering translation direction, In Spatial Vision in Humans and Robots, Harris L and Jenkin M, eds, Cambridge University Press, p. 39-62, 1993.
Heeger DJ, Normalization of cell responses in cat striate cortex, Visual Neuroscience, 9:181-198, 1992.
Abstract: Simple cells in striate cortex have been depicted as halfwave-rectified linear operators. Complex cells have been depicted as energy mechanisms, constructed from the squared sum of the outputs of quadrature pairs of linear operators. However, the linear/energy model falls short of a complete explanation of striate cell responses. In this paper, I present a modified version of the linear/energy model in which striate cells mutually inhibit one another, effectively normalizing their responses with respect to stimulus contrast. This paper reviews experimental measurements of striate cell responses, and shows that the new model explains a significantly larger body of physiological data.
Heeger DJ, Half-squaring in responses of cat striate cells, Visual Neuroscience, 9:427-443, 1992.
Abstract: Simple cells in striate cortex have been depicted as rectified linear operators, and complex cells have been depicted as energy mechanisms (constructed from the squared sums of linear operator outputs). This paper discusses two essential hypotheses of the linear/energy model: (1) that a cell's selectivity is due to an underlying (spatiotemporal and binocular) linear stage; and (2) that a cell's firing rate depends on the squared output of the underlying linear stage. This paper reviews physiological measurements of cat striate cell responses, and concludes that both of these hypotheses are supported by the data.
Simoncelli EP, Freeman W, Adelson EH, and Heeger DJ, Shiftable multi-scale transforms, IEEE Transactions on Information Theory, 38:587-607, 1992.
Abstract: Orthogonal wavelet transforms have recently become a popular representation for multi-scale signal and image analysis. One of the major drawbacks of these representations is their lack of translation invariance: the content of wavelet subbands is unstable under translations of the input signal. Wavelet transforms are also unstable with respect to dilations of the input signal, and in two dimensions, rotations of the input signal. We formalize these problems by defining a type of translation invariance that we call "shiftability". In the spatial domain, shiftability corresponds to a lack of aliasing; thus, the conditions under which the property holds are specified by the sampling theorem. Shiftability may also be considered in the context of other domains, particularly orientation and scale. We explore "jointly shiftable" transforms that are simultaneously shiftable in more than one domain. Two examples of jointly shiftable transforms are designed and implemented: a one-dimensional transform that is jointly shiftable in position and scale, and a two-dimensional transform that is jointly shiftable in position and orientation. We demonstrate the usefulness of these image representations for scale-space analysis, stereo disparity measurement, and image enhancement.
Heeger DJ and Jepson A, Subspace methods for recovering rigid motion I: Algorithm and implementation, International Journal of Computer Vision, 7:95-117, 1992.
Abstract: As an observer moves and explores the environment, the visual stimulation in his/her eye is constantly changing. Somehow he/she is able to perceive the spatial layout of the scene, and to discern his/her movement through space. Computational vision researchers have been trying to solve this problem for a number of years with only limited success. It is a difficult problem to solve because the optical flow field is nonlinearly related to the 3D motion and depth parameters.
In this paper, we show that the nonlinear equation describing the optical flow field can be split by an exact algebraic manipulation to form three sets of equations. The first set relates the flow field to only the translational component of 3D motion. Thus, depth and rotation need not be known or estimated prior to solving for translation. Once the translation has been recovered, the second set of equations can be used to solve for rotation. Finally, depth can be estimated with the third set of equations, given the recovered translation and rotation.
The algorithm applies to the general case of arbitrary motion with respect to an arbitrary scene. It is simple to compute, and it is plausible biologically. The results in this paper demonstrate the potential of our new approach, and show that it performs favorably when compared with two other well known algorithms.
Simoncelli EP, Adelson EH, & Heeger DJ, Probability distributions of optical flow, in Proceedings of Computer Vision and Pattern Recognition, p. 310-315, 1991.
Abstract: Gradient methods are widely used in the computation of optical flow. We discuss extensions of these methods which compute probability distributions of optical flow. The use of distributions allows representation of the uncertainties inherent in the optical flow computation, facilitating the combination with information from other sources. We compute distributed optical flow for a synthetic image sequence and demonstrate that the probabilistic model accounts for the errors in the flow estimates. We also compute distributed optical flow for a real image sequence.
Freeman W, Adelson EH, and Heeger DJ, Motion without movement, Computer Graphics, 25:27-30, 1991.
Abstract: We describe a technique for displaying patterns that appear to move continuously without changing their positions. The method uses a quadrature pair of oriented filters to vary the local phase, giving the sensation of motion. We have used this technique in various computer graphic and scientific visualization applications.
Heeger DJ, Nonlinear model of neural responses in cat visual cortex, in Computational Models of Visual Processing, Landy M and Movshon JA, eds, p. 119-133. MIT Press, 1991.
Heeger DJ and Jepson A, Visual perception of three-dimensional motion, Neural Computation, 2:127-135, 1990.
Abstract: As an observer moves and explores the environment, the visual stimulation in his eye is constantly changing. Somehow he is able to perceive the spatial layout of the scene, and to discern his movement through space. Computational vision researchers have been trying to solve this problem for a number of years with only limited success. It is a difficult problem to solve because the relationship between the optical-flow field, the 3D motion-parameters, and depth is nonlinear. We have come to understand that this nonlinear equation describing the optical-flow field can be split by an exact algebraic manipulation to form three sets of equations. The first set relates the image velocities to the translational component of the 3D motion alone. Thus, the depth and the rotational velocity need not be known or estimated prior to solving for the translational velocity. Once the translation has been recovered, the second set of equations can be used to solve for the rotational velocity. Finally, depth can be estimated with the third set of equations, given the recovered translation and rotation. The algorithm applies to the general case of arbitrary motion with respect to an arbitrary scene. It is simple to compute, and it is plausible biologically.
Heeger DJ, Optical flow using spatiotemporal filters, International Journal of Computer Vision, 1:270-302, 1988.
Abstract: A model is presented, consonant with current views regarding the neurophysiology and psychophysics of motion perception, that combines the outputs of a set of spatiotemporal motion-energy filters to estimate image velocity. A parallel implementation computes a distributed representation of image velocity. A measure of image-flow uncertainty is formulated; preliminary results indicate that this uncertainty measure may be used to recognize ambiguity due to the aperture problem. The model appears to deal with the aperture problem as well as the human visual system since it extracts the correct velocity for some patterns that have large differences in contrast at different spatial orientations.
Heeger D, Model for the extraction of image flow, Journal of the Optical Society of America A, 4:1455-1471, 1987.
Abstract: A model is presented, consonant with current views regarding the neurophysiology and psychophysics of motion perception, that combines the outputs of a set of spatiotemporal motion-energy filters to extract optic flow. The output velocity is encoded as the peak in a distribution of velocity-tuned units that behave much like cells of the middle temporal area of the primate brain. The model appears to deal with the aperture problem as well as the human visual system since it extracts the correct velocity for patterns that have large differences in contrast a different spatial orientations, and it simulates psychophysical data on the coherence of sine-grating plaid patterns.