Photo by Daniel Dubois

A participant viewing different elements in a pattern while his brain is being scanned.

Brain mapping advance takes a first step toward visual “mind reading”

By Melanie Moran and David F. Salisbury
Published: July 22, 2005

A pair of psychologists have shown that it is possible to tell what a person is looking at by monitoring the activity in his or her brain.

“Depending on what they focus their mental resources on, we can read out the content of their visual experience,” says Assistant Professor of Psychology Frank Tong, who performed the research with Yukiyasu Kamitani at the ATR Computational Neuroscience Laboratories in Kyoto.

“This is an important step towards eventually being able to read out the details of what a person is thinking by measuring their brain activity. Full-blown mind reading will be a ways coming, but this is a first step.”

The researchers were able to do this by applying a new statistical technique to a standard brain-scanning technique called functional Magnetic Resonance Imaging (fMRI). It is a special type of MRI technology that detects the various areas in the brain that become active during different mental tasks by registering variations of blood-and-oxygen flow.

	Courtesy of ATR Computational Neuroscience Labs
	Yukiyasu Kamitani

Tong and Kamitani’s findings, published in the May 2005 issue of Nature Neuroscience, also suggest that the early stages of image processing in the cerebral cortex may be linked directly to conscious perception and experience. “The brain areas that represent the physical features of the world seem to be the same areas that represent these features in our consciousness,” Tong says.

Tong and Kamitani used the fMRI scanner to observe activity in the primary visual cortex, or V1. This region, located at the back of the brain, receives input from the retina via a way-station in the thalamus. V1 is the area in the cerebral cortex where visual information is initially processed. It contains neurons that burst into activity when lines or edges are perceived. The neurons are organized into columns, each of which responds to different angles of orientation. The individual columns are just fractions of a millimeter wide, too small to be viewed directly using fMRI. But the statistical method that Tong and Kamitani developed allows them to infer the orientation of the lines in a visual pattern that is stimulating the V1 neurons.

Courtesy of Frank Tong
When subjects looked at a plaid image and were asked to pay attention to the left-tilted or right-tilted lines, activity in the visual cortex closely mirrored the person’s mental state. The colored patches show the subtle orientation preference of each 3x3x3 millimeter patch of brain. These brain activity patterns were then analyzed by a computer. By reading out what activity patterns were evoked, the researchers could accurately determine what image was dominant in the person’s mind, even when both patterns were being shown at the same time.

“Visual areas of the human brain are exquisitely sensitive to subtle differences in the angle of lines, and by processing the angles of many lines across a visual scene, people are able to pick out contours, shapes and outlines that define objects,” Tong says. “With our analysis, we are able to read out subtle differences in the angle of lines that a person is seeing.”

To test the procedure, the scientists showed research subjects a simple plaid pattern of crossed lines and asked them to focus on one set of lines by paying attention to small changes in the width of the bars. When they applied their statistical methods to the data, they confirmed that they could accurately identify which of two sets of lines an individual was concentrating on while viewing the same plaid images.

In the past, psychologists thought that such detailed mapping of brain activity was possible only through experiments that use electrodes implanted directly in the brain. The invasive nature of these experiments, however, has prevented scientists from performing them in humans, slowing attempts to understand the precise workings of human brain in many areas, including perception. As a result, the use of non-invasive fMRI to produce data at such a level of detail opens up significant new avenues of research into how humans see and perceive their world.

Courtesy of Frank Tong

This image illustrates the pattern of lines subjects were shown. The solid line shows the actual orientation and the gray line shows the orientation predicted by a computer algorithm that analyzed the person’s brain activity. The researchers were able to accurately read out which of the eight patterns the subject was seeing, just from measurements of that person’s brain activity.

“No one would have thought this was possible a few years ago,” Tong says. “As we develop better ways to measure brain signals with more precision, we can begin to understand what a person is perceiving even if they cannot communicate it.”

Photo by Daniel Dubois
Frank Tong and research assistant Emma Ferneyhough make last-minute adjustements before a brain-mapping session.

The research was supported by grants from the National Institutes of Health and from the Japan Society for the Promotion of Science and National Institute of Information and Communications Technology.

Frank Tong's Online CV

Tong Lab website

Yukiyasu Kamitani's home page

Nature Neuroscience News and Views story commenting on the study