Perception Lecture Notes: Attention and Awareness

Professor David Heeger

What you should know about this lecture

Hemifield Neglect

Parietal lobe lesions (e.g., due to stroke) cause deficits in atention and spatial operations. Deficits are most severe when the lesion is in the right hemisphere and most severe just after a stroke - one typically gets a bit better over time.

The symptoms include:

Summary: These patients suffer from loss of some form(s) of awareness, particularly having to do with the relationship between their own body and the visual world around them. There is lots of evidence that the parietal lobe is involved in "sensory-motor" integration, turning perceptual information into motor action. That's why some of these symptoms show a loss of awareness of the visual world and some of the symptoms show a loss of body awareness.

Patients with temporal lobe lesions (causing face and object agnosia) are fully aware that there's a problem and they develop strategies to compensate for it.  Parietal patients are often unaware of their deficits and confabulate (denial) when forced to confront it.

In hemi-field neglect, loss of attention and awareness can result in devastating behavioral deficits.  However, there are some equally striking examples of residual behavioral performance in the absence of awareness.

Action without Awareness

Blindsight: There are two particularly famous subjects: GY and DB. Each had brain damage to V1 in one hemisphere that left the subjects totally blind (no visual awareness) in large parts of the contralateral visual hemifield.

The black region marks the blind region of the visual field. One has the patient do a series of forced-choice trials. One presents a light on half the trials, and forces the subject to guess on each trial whether or not the light has been presented. Numbered insets indicate percent correct. There is near-chance (43% correct) performance when the light was flashed in the blind spot (labeled "Disc"). However, performance was well above chance for all other positions within the blind region. As far as the subject is concerned, none of the lights were seen, although for some positions he had a non-visual feeling that something happened (hatched area). This non-visual awareness of a visual stimulus has been described by subject GY as "The nearest I ever get, and it is not a fair comparison, is waving your hand in front of your eyes when they are closed. You are kind of aware that something happened but you don't quite see it."

Split brain patients: Neurosurgeons sometimes cut the corpus colossum (the massive bundle of nerve fiber axons connecting the two hemispheres) to treat epilepsy. Such an operation results in subtle, but very interesting deficits.  One of the most interesting effects is another example of residual behavioral performance without awareness.

Language is centered (in most right-handed subjects) in the left hemisphere. The subject fixates the center point, a word flashes up on one side or the other. The subject reports (through the speaking left hemisphere) only the words flashed to the right visual hemifield and denies seeing the left field stimuli. Even so, when instructed to pick up the object corresponding to the flashed word, the left hand correctly retrieves objects for which the subject verbally denies having any knowledge. Sperry won the Nobel prize in 1981 for work on split-brain patients

Consciousness, what is it good for? Blindsight patients' residual vision allows for rudimentary discriminations for action, but they can't use their residual capacity for "thinking" or "imagery" (mental manipulation of a percept). They can't, for example, compare what they are looking at now with what they saw yesterday (because they don't "see" what they are looking at now).

Visual Attention

Eye/head movements:  You are always moving your eyes around to foveate key visual features.

This is the most direct way to shift attention, called overt attention.  Poor resolution in the periphery means that you are aware primarily of things near the center of gaze. There are lots of things you can't do without moving your eyes (for example: count a bunch of briefly flashed dots, even if they're bright enough to give a strong afterimage)

Covert attention: You can also shift attention without eye/head movements to "filter out" unattended locations.

In a typical covert attention experiment, subjects perform a difficult (threshold) visual discrimination task. There is a cue just before each trial that indicates where they should focus their attention. Sometimes the cue is correct, sometimes it is misleading (it cues the wrong location). Performance is often measured as reaction time, but sometimes (as I would prefer) as d' (discriminability via signal detection theory). On average, performance is better on trials that are cued correctly with enough lead time. Here is a summary of the results of this sort of experiment:

  1. You can shift spatial attention without eye movements.
  2. The shifts can be controlled by the subject's decision to allocate attention (e.g., with instructions, or with a central arrow as a cue) or by automatic attraction (e.g., if something suddenly appears in the periphery of your visual field).
  3. Performance falls off smoothly with distance when cued to the wrong location.
  4. Automatic shifts of attention (e.g., caused by sudden appearance of an object in the periphery)are linked in time to the occurrence of the cue. This yields best performance right after (50-300 msec) the cue, then performance is worse if there is a long delay (> 300 msec) between cue and target. The drop-off in performance for long delays is sometimes called inhibition of return.
Visual search: Some patterns are easily discriminable (pre-attentive). But others are not, and require something like a serial/scanning search of the visual stimulus.

In this example of texture segmentation, it is easy to see the pluses on the background of L's, but to see the region of T's is hard, and requires item by item scrutiny (i.e., allocation of visual attention).

It is hard to search for a vertical rectangle among horizontal rectangles and ovals of both orientations. It would be easy to search for a vertical rectangles if the only distractors were horizontal rectangles. The conjuction of two features (vertical and rectangle) make it hard.  This visual search task is like the "Where's Waldo" books.

Difficult discriminations require attention - spotlight or zoom lens, and item-by-item serial search. They show a set-size effect whereby reaction time is increased linearly with an increase in the number of distractors (non-target items). Not only that, this increase for each irrelevant item is twice as big, on average, on trials where the target is absent (and the observer should say "no") than on trials where the target is present (and the observer should say "yes"). This is because the observer needs to attend every item to be assured that "no" is the correct answer, but on average will only have to look at half the items to find the target, when it is indeed present.

Feature integration theory (Treisman): This is a theory for explaining which patterns are easily discriminable and why.  The idea is that the front-end of the visual system breaks the stimulus down into its constituent parts. It separately analyzes each local patch of the visual field to determine: pattern; motion; shape (depth and size); color, etc. Attention is the glue in feature integration theory. Example: seeing red is preattentive, and seeing rightward motion is preattentive, but seeing a red thing moving to the right requires attention to connect the red thing with the moving thing. The theory does not go so far as to say how attention accomplishes this gluing. Feature integration theory is based in part on the functional specialization hypothesis, e.g., that "red" and "moving rightward" are represented in separate visual brain centers. One needs attention to bring those separate neuronal representations together.

Illusory conjuction: Suppose we briefly flash a picture of woman with black hair and a red sweater. Some subjects will incorrectly report seeing a woman with red hair. This sort of "illusory conjunction" is taken as evidence for feature integration theory. In a brief display, there's not enough time for the attentive processing to combine the features correctly.

Change Blindness

Change blindness is a phenomenon in which very large changes occurring in full view in a visual scene are not noticed. Look for the change that is occuring between the two frames of the movie above. What characterizes this and other change blindness demonstrations is the fact that the changes are arranged to occur simultaneously with some kind of extraneous, brief disruption in visual continuity, such as the large retinal disturbance produced by an eye saccade, a shift of the picture, a brief flicker, a "mudsplash", an eye blink, or a film cut in a motion picture sequence. These phenomena are attracting an increasing amount of attention from experimental psychologists and from philosophers, because they suggest that humans' internal representation of the visual world is much sparser than usually thought. It is as close as you can get (as a normal observer) to the experience of neglect patient with a parietal lobe stroke.

Attention in the Brain

Goldberg & Wurtz (NIH) were the first to demonstrate that attention can affect neural responses. They recorded from neurons in the parietal lobe in awake monkeys while the monkeys were performing various tasks that they had been trained on.

Top: The monkey was trained to fixate. There is a small response from the parietal lobe neuron when a light comes on in the periphery.

Middle: The monkey was trained to fixate until the peripheral light comes on, then it must move its eyes to look at the light. The retinal stimulation is identical, but there are much bigger responses (3x).

Bottom: The monkey was trained to move its arm instead of its eyes. One gets the same result (bigger response), i.e., this is not simply an eye movement control signal.  It is rather more like the neural correlate of attention, salience or task-relevance.

The conclusion of this study and many more like it is that visual responses in cells in the secondary cortical visual areas are gated according to the behavioral significance of the stimulus. There are neurons in the brain that are correlated with attention. When a visual stimulus is relevant for the task at hand, those neurons are more active.

Attentional modulation of V1 brain activity: The classical (Hubel and Weisel) view of V1 is that it acts as a passive, automatic, image processing machine. An emerging view is that V1 does much more than that.

Here is an example of an fMRI experiment performed in Prof. Heeger's lab. Subjects viewed moving stimuli presented within a pair of apertures, one positioned to the left and one to the right of the center of fixation. The shape of the fixation point cued subjects to attend alternately to a series of motion discrimination trials on the right and to a series of trials on the left. Because the discrimination was difficult, subjects had to allocate spatial attention to the relevant (cued) side of the visual field in order to perform as well as possible. V1 brain activity, measured with fMRI, modulates as subjects alternate attention. The "Attend left" condition increases brain activity in the right hemisphere. The "Attend right" condition increases activity in the left hemisphere.  Attention reaches all the way back to the first visual area in the cortex.

Why not attend to everything all the time?

Mental Imagery

Consciousness is associated with the capacity for "thinking" or "imagery" (mental manipulation of the percept). What happens in the brain when you simply imagine a visual stimulus? Imagine a waterfall - it is a similar perceptual experience to actually looking at a waterfall - does it evoke activity in MT? Imagine Bill Clinton's face - does that evoke activity in the "face" area? Recent fMRI experiments suggests that imagery and perception result in a similar pattern of brain activity.

Nancy Kanwisher (at MIT) performed an fMRI experiment to measure mental imagery in the brain. Once every 12 seconds, subject heard the name of a person or a familiar place (a building on the MIT campus). They were instructed to imagine it. The investigator recorded fMRI signals in the "face" area (that responds strongly to pictures of faces) and the "place" area (that responds strongly to pictures of familiar places, buildings, etc.).  The red curve shows the time-course of response in the "face area". The blue curve shows the time-course of response in the the "place area". fMRI response is bigger in the "face" area when imagining a face. Response is bigger in the "place" area when imagining a place.

Scientific Study of Consciousness

What is Consciousness?
  1. Not all operations of the brain correspond to conscious processes (Helmholtz's unconscious inference).
  2. Consciousness is closely associated with attention. William James (1890, influential psychologist, brother of novelist Henry James), "Everyone knows what attention is. It is the taking possession by the mind, in clear and vivid form, of one out of what seem several simultaneously possible objects or trains of thought... It implies withdrawal from some things in order to deal effectively with others."
  3. Consciousness involves some form of memory. Charles Richet (1884) wrote, "to suffer for only 1/100th of a second is not to suffer at all; and for my part I would readily agree to undergo a pain, however acute and intense it might be, provided it should last only a hundredth of a sec, and leave after it neither reverberation nor recall."

We have now seen several examples of how brain activity is linked with perception, even in cases for which the percept is entirely subjective/illusory:

  1. Brain activity in V1 fluctuates with perceptual alternations during binocular rivalry.
  2. Microstimulation in MT biases monkey's judgements of motion.
  3. Brain activity in MT is correlated with the perception of illusory motion during the motion aftereffect.
  4. Neural responses in V2 are correlated with the perception of  illusory contours.
  5. Brain activity in many visual areas including V1 modulates with shifts in attention.
  6. Imagining different attributes of visual stimuli (e.g., color, motion, faces) selectively activates the specific brain pathways involved in the perceptual processing of those stimulus attributes.
Here is a bold, unsubstantiated claim (revisited): Your perceptual experiences are no more than the activity of neurons in the sensory areas of your brain. Different brain areas are involved in different aspects of perception.  Eventually, we will know enough about this so that if you could measure the pattern of activity in the appropriate brain areas of my brain, you would be able to accurately predict what I am perceiving.

We do not yet understand specifically what distinguishes neural activity that underlies conscious (versus unconscious) mental states. There are several hypotheses (some of which are listed above) but none of them yet have strong empirical support.

For decades, neuroscientists have worked toward simplification, using simple stimuli and behavioral tasks, precisely parameterized and in highly controlled laboratory settings. This approach has obvious advantages and has served us well, as evidenced by the tremendous amount of knowledge amassed about brain structure and function. These conditions, however, are removed from natural real life situations. Recently, some neuroscientists have been exploring the function and organization of the human brain under more natural and unbounded settings. In one experiment, brain activity was measured using functional magnetic resonance imaging (fMRI) during free viewing of an engaging sensory and emotional experience (a Clint Eastwood movie). This research has revealed a surprising tendency for individual brains to “tick collectively” during natural sensory and emotional experiences. About 40% of each individual brain (yellow and orange in the above pictures) does the same thing as other brains when watching the same movie. Moreover, the results demonstrated that the unified nature of conscious experience in fact consists of temporally interleaved and highly selective activations in an ensemble of specialized brain regions, each of which “picks-up” and analyzes its own unique subset of information according to its functional specialization. One area of the brain, for example, responds every time there is a close up of a face. Another area analyzes outdoor scenes and landscapes. And another area responds when viewing fine hand motion manipulations. Etc.

To find out more about current scientific research on consciousness, go the Association for the Scientific Study of Consciousness web page


Copyright © 2006, Department of Psychology, New York University
David Heeger