Photo by Neil Brake

Thomas James

Seeing is believing...
even when it's ambiguous or misleading

By David F. Salisbury
Published: July 19, 2004

Seeing is believing, even when it's ambiguous or misleading.

New research has found that, when judging an object's motion, the brain continues to accept ambiguous visual information even when it conflicts with more reliable tactile input. The studies, which appear in the June 7 issue of the journal Psychological Science and a forthcoming issue of the journal Cognitive, Affective, & Behavioral Neuroscience, provide new insights into the way in which the brain blends and balances information from different senses in its constant effort to comprehend the external environment.

Photos by Neil Brake and Daniel Dubois
Randolph Blake, left, and Thomas James

Scientists have long known that vision is the dominant sense in humans and other primates. But evidence is growing that there is greater sharing of information than they had thought between the visual system and the somatosensory system, which includes a variety of bodily senses, including touch, pain, pressure, temperature, and joint and muscle position sense. In fact, mounting evidence suggests that regions of the brain that have been considered exclusively visual in nature may process somatosensory input as well.

The research, which was conducted by a team of Vanderbilt psychologists – Centennial Professor of Psychology Randolph Blake and research associates Thomas W. James and Kenith V. Sobel – found that the region of the brain – the middle temporal visual center, MT – that specializes in processing visual movement also responds to motion detected by touch. However, they were surprised to discover that, when presented with ambiguous visual information and reliable tactile information, the brain did not fuse the two into a single, accurate representation of motion as the prevailing theories predicted. Instead, they found that it keeps the two inputs separate, accepting a degree of “cognitive dissonance” when the two conflict.

"This suggests that there is naturally a higher level of inconsistency between seeing something moving and feeling something moving than there is between seeing and feeling something's shape,” says James. “In that case, our perceptual system should combine motion and shape information differently. If the two sources of motion information are often inconsistent, then it is better if the brain is not obliged to fuse them.”

The Vanderbilt team decided to investigate the relationship between vision and touch because the two senses share a special relationship to shape. Although eyeballing an object and exploring it with one's fingers are distinctly different experiences, both can provide the brain with detailed information about an object's shape in three dimensions. This is different from temperature, for example, because temperature cannot be determined visually, only through touch. On the other hand, touch cannot distinguish an object's color, only vision can.

In previous experiments the researchers had studied how the brain uses vision and touch to identify shapes. So they decided to explore a different characteristic: motion. Although this was considerably more difficult to study than shape, they came up with procedure that relies on a well-known visual illusion called the kinetic depth effect.

Courtesy of Psychological Science
Illustration of experimental setup

The kinetic depth effect begins with a group of dots arranged in a circular pattern on a computer screen. The dots are programmed so that half move left to right and half move right to left. The dots' movements are choreographed so that they move as if they are fixed to the surface of a transparent globe. This tricks the brain into seeing them as forming a rotating, three-dimensional sphere. However, the illusion doesn't give the brain any clues about the direction that the sphere is rotating. So the brain splits the difference and the sphere appears to rotate from right-to-left 50 percent of the time and from left-to-right 50 percent of the time. Although the rate at which the illusory sphere switches direction varies from person to person, the 50/50 split in duration is universal.

To use this illusion, the researchers set up an experimental environment with a rotating Styrofoam ball at a distance where subjects could touch it with two hands. Using a pair of mirrors and computer displays, they projected the moving dot illusion described above into the subject's eyes in such a way that it appeared to be the same size, rotate at the same speed and occupy the same position as the physical ball.

	Courtesy of Psychological Science
	The bar graph shows the average responses of five observers. Red bars show the time during which the tactile globe was rotating in the same direction as the visual globe. Green bars depict the time during which the tactile globe rotated in the opposite direction. The yellow bars give the average length of the stationery trials.

“Our thought was that if a person touched something turning in a specific direction then the perceptual system should fuse the tactile and visual information, resulting in people ‘seeing' a sphere rotating in a direction that was consistent with the more reliable source of sensory information,” says James.

Instead, they found that the tactile and visual information did not fuse, but the tactile information did have an effect. The subjects reported that the time the visual sphere spent rotating in the same direction as the physical sphere jumped from 50 percent to 65 percent, yet it continued to appear to rotate in the opposite direction 35 percent of the time.

The researchers went a step further and monitored the brain activity that was taking place using functional magnetic resonance imaging (fMRI), a technique that detects activity levels in different part of the brain by measuring blood flow. They knew from previous studies that the part of the brain most likely to be involved was the Middle Temporal visual center (MT). They recorded a reliable increase in activity in MT when the globe was rotating compared to when it was stationary for both vision and touch. But the activity stimulated by the visual stimulus was four times greater than that generated by touch.

In the Psychological Science article, the researchers point out that people are remarkably adept at judging the size, shape and mass of objects by touch alone and report that their experiment shows that high-fidelity touch information can also influence visual perception of three-dimensional motion.

But a central question remained unanswered: Did the visual and tactile information fail to fuse because the brain could distinguish between the physical and the visual spheres? The fact that subjects could not see their fingers touching the rotating sphere might have been enough to keep the fusion to take place, the researchers speculated.

Courtesy of CABN and Thomas James
Illustration of the experimental setup and video of the version of the kinetic depth effect achieved by viewing a rotating wire sphere with one eye.

So they came up with another approach, which is described in the Cognitive, Affective, & Behavioral Neuroscience article . They made a wire-frame sphere and attached it to a motor so it rotates slowly. When viewed from a distance with only one eye, this produces an illusion comparable to that created by the moving dots. It appears to rotate first in one direction and then in another. In this case, however, subjects can reach out and directly touch the sphere to determine the direction that it is rotating.

	Courtesy of CABN
	The two graphs show the percentage of the time that participants reported that the wire sphere appeared to be rotating in the direction consistent with its physical rotation. The upper graph shows the results when the participants were not touching the sphere, touching it with a thumb and enclosing the sphere with their hand. The lower graph shows how their perception differed when touching the sphere with one, two or three fingers.

Once again, the researchers were surprised to find that this procedure produced almost precisely the same results as their first effort. Without touching the sphere, subjects reported that it appeared to rotate in each direction approximately 50 percent of the time. When the subjects were touching the sphere, it appeared to rotate in the direction consistent with touch 60 to 70 percent of the time but still appeared to rotate in the opposite direction 30 to 40 percent of the time.

Unlike the initial experiment, subjects also reported a significant level of “perceptual dissonance” when the direction of rotation that they saw was inconsistent with that which they felt. (They described this feeling with terms such as “weird” and “freaky.”)

“It is surprising that an unreliable visual stimulus should be that resistant to tactile input,” says James. This points out that, in terms of visual motion processing, the mechanism that produces alterations in perceptual state must be very powerful because it is not over-ridden even by highly reliable tactile input.”

American Psychological Society news release: Vision's Touchy-Feely Side

Thomas W. James' home page

Randolph Blake's home page