Monday, July 01, 2002

Touching Tastes, Seeing Smells—and Shaking Up Brain Science

“With an intense flavor,” said one man, “a feeling sweeps down into my hand, and I feel weight, texture, shape...” Neurologist Cytowic, author of The Man Who Tasted Shapes, revisits one of the brain’s most intriguing anomalies. People with synesthesia may hear colored sounds or even taste shapes, because the joining of two senses runs through almost every possible combination. New research suggests that what was once seen as merely a bizarre disorder raises knotty questions for brain science and may offer clues to the brain’s mysteries. “One cannot admit a wrecking ball,” writes Cytowic, “and expect the house to remain standing.”

Sometimes the bizarre is just the bizarre, but when the anomaly is a product of fundamental brain processes yet seems to confound them, the theory makers have a problem. For several decades, most neurologists who heard about synesthetes—people who see colored letters, feel colored pain, and taste shapes—just shrugged their shoulders or rolled their eyes. -

Now, as evidence accumulates that synesthesia is a bonafide neurological phenomenon, some are asking how this squares with some of the most cherished conceptions of contemporary brain science. The good news is that when old theoretical structures fall, new light may flood in.

Some 20 years ago, my dinner host, Michael Watson, delayed our seating by announcing, “There aren’t enough points on the chicken.” He meant that the taste still failed to evoke the prickly sensation he sought. To Michael, tastes and smells were also felt as a physical touch in his face and hands. “With an intense ﬂavor,” he tried to explain, “a feeling sweeps down into my hand, and I feel weight, texture, shape, and whether it’s hot or cold as if I’m actually grasping something.”

“Ah,” I exclaimed, “You have synesthesia.” Michael looked stunned. “You mean there’s a name for this thing?”

Sharing a Greek root with anesthesia, which means “no sensation,” synesthesia means “joined sensation,” wherein two or more senses are coupled in such a way that a voice, for example, is not only heard but also felt, seen, or tasted. We call individuals with this coupling synesthetes.

The process of synesthesia usually travels in only one direction. For example, a sweet taste made Michael feel a cool, polished, curved surface in his hands, but handling a billiard ball would not elicit any flavor in him. About 40 percent of synesthetes have multiple types of synesthesia; the couplings that have been observed by scientists do not include all possible pairings. Sensing letters, numbers, or words as colored accounts for two-thirds of the instances of synesthesia. For those with “colored hearing,” sounds evoke colored shapes that arise, move, alter, and fade—somewhat like fireworks. For others, merely thinking about a certain person, object, or concept evokes synesthetic sensations.

As children, synesthetes are surprised to discover that others do not share these experiences. Often ridiculed and disbelieved, they learn to keep their atypical perceptions to themselves. Nonetheless, the phenomenon remains involuntary and consistent throughout their lives. The trait runs strongly in families, and the genetics of its inheritance are reasonably well understood. Some type of synesthetic experience occurs in perhaps 1 in 200 individuals, and more than 75 percent are women.

Like most anomalies that lie outside the explanation of conventional theories, synesthesia was long dismissed as a mere curiosity or, worse, as just subjective imagination. I wondered how the brains of people like Michael Watson might differ from the majority, but my colleagues refused to accept that his experience might be a bona ﬁde neurological phenomenon. Pointing to 200 years of synesthesia history in the annals of medicine and psychology did not sway them. Today, however, researchers in some 15 countries are studying synesthesia, and many doctoral candidates have chosen it for their theses.

As children, synesthetes are surprised to discover that others do not share these experiences. Often ridiculed and disbelieved, they learn to keep their atypical perceptions to themselves.

If others have gradually come to accept the reality of synesthesia, they must now relinquish some received wisdom about how the brain works. Our concepts of how things work are but models, after all, reductions of reality that arise from human minds; history has shown repeatedly that reality has a way of making a mess of neat and tidy concepts. Like most exceptions of nature, synesthesia is forcing a paradigm shift. One cannot admit a wrecking ball and expect the house to remain standing. Paradoxically, the very thing that destroys simultaneously illuminates, and what emerges may surprise us.

SUBJECTIVE REALITY OR POPPYCOCK?

In 1989, I reported my initial studies on several dozen synesthetes in Synesthesia: A Union of the Senses. Here I proposed that the phenomenon pointed to deep cognition, meaning fundamental processes that underlie how we perceive and think. It was “a voice in the wilderness,” to quote one of today’s researchers, V. S. Ramachandran. In fact, 11 years before my own effort, Lawrence Marks, a psychophysicist at Yale, suggested in The Unity of the Senses that understanding synesthesia might shed light on the perceptual basis of metaphor and perhaps even the acquisition of language itself. He, too, was mostly ignored.

Often throughout its history, synesthesia had been dismissed simply because the condition is revealed only through an individual’s self-reported mental state. There is no test for it in the usual sense of that word. The complaint that introspection is inherently unreliable and therefore impermissible as scientiﬁc data has a long history. In the 19th century, psychophysicists such as Gustav Fechner tried to formulate laws regarding sensation and perception based on observers’ reports, taking as a given that mental states exist. Scientists in the 20th century, however, consistently strove to eliminate the subjective role of a human observer in gathering empirical data. Within psychology, the triumph of behaviorist theory further ensured that inquiry into mental life would remain taboo for decades.

Like most exceptions of nature, synesthesia is forcing a paradigm shift. One cannot admit a wrecking ball and expect the house to remain standing.

Because a technological focus dominated science in general and medicine in particular, my neurology colleagues unsurprisingly asked what Michael’s CAT scan showed. In questioning synesthesia’s reality, they sought a third-person technological veriﬁcation of a ﬁrst-person experience. Technical corroboration is one thing; but the sweeping assumption that anyone’s personal experience is invalid is quite another. Even current functional brain imaging, which is supposed to be anatomically objective, starts with what one wants to verify objectively: the subject’s state of mind.

Synesthesia refuses to be ignored, affirming loudly that subjective mental worlds do exist. Among other things, therefore, synesthesia grants us an opportunity to examine the dichotomy of objective-subjective experience. But its importance goes deeper than that.

A LINK WITH THE PAST

Compared with the hostility of modern objectivists, a fair number of earlier scientists accepted synesthesia as a genuine phenomenon after Sir Francis Galton’s 1880 report in Nature on “visualized numerals”—if only because the individual stories sounded so similar, giving it the clinician’s feel of a genuine phenomenon. The earliest medical reference, a case of sound-induced color, dates to 1710, but the style and details of Galton’s report make his the first recognizably modern one. For example, Galton’s synesthetes express astonishment at discovering that they are unusual. Most claim to have had the ability as far back as they can remember and, far from trying to appear special or call attention to themselves, genuine synesthetes prefer to hide their trait because of the ridicule they suffer upon disclosure.

The experience of synesthesia is difﬁcult to express, as witnessed by the collective struggle to convey exactly what is sensed. Even computer animations are said to be only about 60 percent representative of “what it is really like.” As Galton noted in his 1883 Inquiries into Human Faculty, those with visual synesthesia are “invariably most minute in their description of the precise tint and hue of the color. They are never satisfied, for instance, with saying ‘blue,’ but take a great deal of trouble to express or match the particular blue they mean.”

Compared with the hostility of modern objectivists, a fair number of earlier scientists accepted synesthesia as a genuine phenomenon.

Indeed, one can take the details of a given synesthete today and find matching examples in the classical scientiﬁc literature, linking the efforts of scientists a century ago with those of contemporary ones. Ironically, it is precisely synesthetes’ subjective claims that now form the basis of today’s experiments that address predictions regarding the trait’s perceptual reality.

WHAT DEFINES SYNESTHESIA?

Synesthesia can be acquired via epilepsy or the ingestion of hallucinogens such as mescaline or LSD, but idiopathic (or developmental) synesthesia arises naturally without an external agent or brain abnormality. There is nothing in need of medical treatment. The subjective, ineffable, and idiosyncratic nature of this kind of synesthesia does make it an easy target for dismissal. Even the term “synesthesia” has been used imprecisely over time, referring to everything from metaphor (loud tie, sharp cheese, sweet voice) to deliberate contrivances such as son et lumière theatrical performances and “smellavision.”

A clear deﬁnition avoids a muddle. Idiopathic synesthesia is defined by five clinical findings: It is (1) involuntary and automatic, (2) spatially extended, (3) consistent and generic, (4) memorable, and (5) affect-laden. These refer to speciﬁc characteristics of the synesthetic person’s experience.

INVOLUNTARY AND AUTOMATIC

Synesthetes claim to hear a certain sound or to look at a letter, for example, and then to see a color. “It just happens,” some say. How can we demonstrate that they have no control over their experiences? Phenomena called “perceptual grouping” and “pop-out” demonstrate that the response is indeed automatic. For example, imagine that a group of 2’s arranged to form a triangle is embedded in an array of 5’s drawn in a way that the ﬁgures resemble mirror images. When told to look for a hidden shape, most of us would take time to hunt down the target triangle buried within the distracting 5’s. But a synesthete who sees every numeral as differently colored would immediately see the target pop out of an alternatively colored background. If the perception is involuntary, synesthetes should perform much faster than nonsynesthetes—and they do.

Two individuals with the same kind of synesthesia will rarely agree as to the particulars of what they perceive. The numeral 2 may be green or red or turquoise for different people.

Because synesthetic associations are idiosyncratic, such tests must be tailored to the individual. That is, two individuals with the same kind of synesthesia will rarely agree as to the particulars of what they perceive. The numeral 2 may be green or red or turquoise for different people. Deliberately inducing mismatches—say, by printing graphemes (a language’s written elements) in ink colors that are either congruent or incongruent with a given synesthete’s perceptions—and then measuring reaction times to them has become a popular approach in current research.

In another setup, surrounding a target grapheme in the visual periphery with other letters renders it “invisible,” meaning that it is not consciously perceived. Remarkably, it still evokes the synesthetic color. “It must be ‘A’ because I see red,” a subject will say. This implies that synesthesia is evoked at an early sensory level—a preconscious one, in fact.

As these examples show, many of the probes designed to reveal whether synesthesia is automatic also turn out to prove that synesthesia is perceptual. What are called random dot stereograms do even more, helping us identify the lowest brain level at which synesthesia can occur. When the left eye looks at one pattern of black dots and the right eye at another, the two images fuse in the brain, causing a three-dimensional object to pop out from the viewing plane. Synesthetes see the object, as everyone else does, but they see it in color. This result says two things: that synesthetic color arises after binocular fusion (setting the lower brain limit above the ﬁrst synaptic level of visual neurons in the cerebral cortex, called V1), and that color appears to be bound to a form as the form is being recognized.

SPATIALLY EXTENDED

Some synesthetes describe Technicolor reading “on the page,” even as they simultaneously see the black ink of the printing. Others with colored hearing speak, for example, of watching “a screen about six inches from my nose.” Michael Watson often reached out in front of him to feel shapes at arm’s length. Even those who say the synesthesia is in their “mind’s eye” remark that it differs from ordinary vision and imagination by its quality of Euclidean locus, meaning that it has a sense of physical place. That is, synesthetes speak of “going to” or “looking at” a certain place to examine a sensation.

Tests of visual perception demonstrate the reality and consistency of synesthetes’ responses. These are examples from synesthetes who see colored numbers—for example 5 is green and 2 is orange. In the top illustration, the black numeral 5 composed of smaller 2’s is seen as green when this synesthete focuses on the large figure but as orange when focusing on its components. In the middle series, a random dot stereogram presents one pattern of dots to the right eye and a different pattern to the left eye. The two images are fused in the brain (“binocular vision”) and, once again, the numeral 2 stands out as orange for the synesthete. At the bottom, a triangle of 2’s is imbedded in a pattern of 5’s; most of us would have to hunt for the triangle, but it instantly pops out in red for this synesthete.

[top, center] Palmeri,TJ, Blake, R, Marois, R, Flanery, MA, and Whetsell Jr., W. “The perceptual reality of synesthetic colors.” Proceedings of the National Academy of Sciences; 99(6): 4127-4131. © 2002, National Academy of Sciences, USA; [bottom] Ramachandran, VS, Hubbard, EM. “Psychophysical investigations into the neural basis of synaesthesia. Proceedings of the Royal Society of London, Biological Sciences 2001; 268(1470): 979-83

This quality of spatial extension is particularly dramatic in the perception of what are called “number forms.” (The term is somewhat of a misnomer, given that number forms concern not just integers but any concept involving serial order.) The perceptual qualities of spatial location, shape, and, often, color become synesthetically joined to semantically ordered concepts such as integers, months, the alphabet, shoe sizes, temperature, and so forth. For example, each day of the week or month of the year may be associated with a different colored shape, which is perceived in a location speciﬁc to the individual. Number forms are usually colored and create circles, zigzags, loops, and various tortured conﬁgurations.

Synesthetes often remark that some colors they see are “weird”— ones that they would never deliberately choose.

Note that we may speak of synesthesia as “joined senses”—a sound being associated with a visual perception, for example— but spatial configuration, letters, words, days of the week, and the like are not senses at all; they are categories of knowledge. Because they reckon among the most frequent manifestations of synesthesia, we need to enlarge our deﬁnition beyond pure sensory-sensory pairings to include the binding of sensory fragments (qualia) to categories of mental concepts. I will return to this later.

CONSISTENT AND GENERIC

Once established in childhood, synesthetic associations remain stable throughout life, as demonstrated by tests and retests spanning many years. For example, synesthetes may be asked to indicate their color responses to a list of words. When tested without warning a year later, they report almost identical responses, whereas controls without synesthesia, even if forewarned of retesting a month before, perform near chance level.

Synesthetes often remark that some colors they see are “weird”—ones that they would never deliberately choose. They may see colors that they do not like or wish that they saw their favorite ones more often. This should not be surprising, given that their visual systems are being stimulated via nonoptical means over which they have no control. In one interesting example, a color-blind synesthete with S-cone deﬁciency—which makes it hard to discriminate blues and purples—speaks of seeing numbers in “Martian colors,” meaning colors he is unable to see in the real world. Curiously, synesthesia happens to be more common in blind individuals than the general population.

Saying that synesthesia is generic, as well as consistent, means that what is experienced is not complex and pictorial, but elementary— blobs, lattices, cold, rough, sour, zigzags, simple geometric shapes, and so forth.

MEMORABLE

When asked what good the trait does, synesthetes immediately answer, “It helps you remember.” They do have measurably high memories, sometimes photographic ones, “eidetic” in psychological parlance. The extra bits of information help synesthetes remember things like telephone numbers and names. As one synesthetic neuropathologist puts it, “I use it…to help me remember correct sequences of numbers, words, phrases, letters, to help me remember names and locations of anatomical structures (especially neuroanatomical structures—you should see the beautiful array of colors in the brain!) and neuropathological classiﬁcations. I could go on and on.”

The memory expert that renowned Russian neuropsychologist Alexander Luria described in The Mind of a Mnemonist possessed a ﬂawless memory because everything he recalled was accompanied by synesthesiae in each of his senses: “I heard the bell ringing... A small round object rolled right before my eyes... My ﬁngers sensed something rough like a rope... then a taste of saltwater... and something white.”

Synesthesia carries a sense of certitude, sometimes a “Eureka!” feeling. Most ﬁnd it highly pleasurable.

AFFECT–LADEN

Synesthesia carries a sense of certitude, sometimes a “Eureka!” feeling. Most ﬁnd it highly pleasurable. Trivial tasks are laden with emotional affect, so that mental calculations are “very pleasurable” and recalling a phone number is “delightful.” Mismatched perceptions can be “like ﬁngernails on a blackboard.”

In a minority of cases, what is perceived is so wretched—for example vile-tasting words, or nausea when playing a musical instrument—that the condition interferes with daily life. Nevertheless, synesthetes say that they would never part with their perceptions. It is hard to overstate the intensity and pervasiveness of affect in synesthesia.

PICTURES, PLEASE

Synesthesia’s reality is demonstrated by its automaticity, consistency, and durability; by its induction of perceptual grouping and pop-out; by the evocation of colors by “invisible” graphemes at an unconscious level; by its strong heritability as an X-linked dominant trait; by the fact that having one type of synesthesia makes one more likely to have a second or third type; and by the ability of color-blind and blind persons to see colors.

Despite these kinds of proofs, some skeptics can be satisﬁed only by machine veriﬁcations that produce pictures of the brain. What is remarkable is how profoundly the emphasis of those pictures has switched from structure to function. When, around 20 years ago, my colleagues asked about Michael Watson’s CAT scan, they expected that a gross brain abnormality must underlie synesthesia if it were real. In other words, where was “the hole in his head”? But given that synesthetes such as Michael are normal, manifesting no evident neurological impairment, a structural lesion such as a stroke, tumor, or a bit of missing brain would be unlikely. As expected, his CAT and MRI scans, which assess structure, were normal. What was wanted was a test of function. -

Artists and composers who are synesthetes often seek to express their unique perceptions in their works. This sculpture by synesthete artist Carol Steen is called Cyto, because it represents for her the shapes and colors of the name [Richard] Cytowic, from whom she first gained knowledge, beyond her own experience, of the widespread phenomenon of synesthesia. “The forms are constructed one on top of the other in a vertical arrangement,” she says, “because I often see flying colored forms appear that way in my synesthetic visions.”

In 1980, I performed the ﬁrst such functional test on a synesthete, using a technique called “regional cerebral blood ﬂow.” This showed that Michael’s brain behaved much differently from nonsynesthetic ones, being strongly perturbed by ordinary stimuli such as smell. This study also conﬁrmed that synesthesia was a phenomenon of the brain’s left hemisphere. This left-brain locus disappoints some people, who want it to be a right-brain function because they consider synesthesia artistic and creative.

In 1995, Eraldo Paulesu and colleagues performed PET scans on six women who saw colors in response to spoken words. PET offers superior spatial resolution and other advantages to assessing function compared with my earlier technique. In this study, spoken words activated auditory and language areas in both synesthetes and controls, but only in the synesthetes did they also activate some visual areas.

Areas of the brain’s visual cortex are labeled according to their primary functions, V1 having to do with sorting the signals for various visual tasks, V2 and V3 relating to the perception of form, V4 relating to color, and V5 relating to motion and direction. Imaging studies show that, surprisingly, synesthetes can generate conscious visual experiences without activating V1 or V2.

Inner Vision: An Exploration of Art and the Brain by Semir Zeki, courtesy of Oxford University Press, 1999

Scientists have labeled only a few of the numerous cortical areas involved in vision using a numbering scheme. V1, formerly called the primary visual cortex, is the first level at which retinal projections synapse in the cortex. V1 acts like a post ofﬁce, sorting and forwarding different kinds of signals to different destinations where different types of transformations are carried out, and so it is expected to activate in all visual tasks. At the second synaptic level, V5 pertains to motion and direction, V4 to color, and V2 and V3 to form perception. At the fourth synaptic level, neither the areas pertaining to facial recognition nor spatial-location encoding has yet received a “V” label. Whereas Paulesu’s study did not show the hoped-for activation of the unique human color area, V4 (probably due to a limitation of the PET technique), it did provide a result that was startling: a failure to activate V1 or V2 in synesthetes. These two early visual areas do activate when control subjects view colors.

This result is inconsistent with a major premise of what is called “blind-sight.” Some brain-damaged patients retain capacities of which they are not conscious. Oxymoronic terms such as “blindsight” or “numbsense” convey how someone unable to see or feel can nonetheless discriminate visual or tactile test targets with high accuracy, despite insisting on not being able to “see” or “feel” anything. Because stricken individuals are oblivious to their unconscious know-how that allows correct discrimination, researchers have postulated that the primary sensory cortex (such as S1, V1, A1), which is damaged in these individuals, is indispensable for any conscious awareness. In the words of Lawrence Weiskrantz, the acknowledged authority in the ﬁeld, “striate cortex [V1] is essential…for any ‘seen’ [consciously experienced] perception whatsoever.”

Not any longer. Synesthetes in the PET study proved that the brain can generate conscious visual experiences without contribution from the primary visual cortex (V1). Blindsight’s implications for consciousness studies therefore need to be rethought. In the meantime, synesthesia supports the claim by vision researcher Semir Zeki that activity in any given module sustaining a given visual function (V4 for color, V5 for motion, V3 for form) is sufﬁcient, as well as necessary, for one to be conscious of that color, motion, or form. That is, activation of V4 alone is sufﬁcient to “see” color, without the necessity of recruiting other visual modules, either upstream or downstream.

BE CAREFUL WHAT YOU WISH FOR

In 2002, a functional MRI (fMRI) study by Julia Nunn and her colleagues at last conﬁrmed what was long expected: V4 activation (without V1 or V2 activity) in synesthetes who see color in response to spoken words. Whereas both synesthetes and controls activated auditory and language areas as expected, the synesthetes also activated the color area (V4), but only on the left—in agreement with earlier results. Such lateralization is tantalizing, given that their color experiences were not conﬁned to the right visual ﬁeld. The fMRI technique, which is the most reﬁned one we have to date, also disclosed activation in areas concerned with memory and emotion, again supporting both the subjective statements and clinical observations of synesthetes.

Synesthetes in the PET study proved that the brain can generate conscious visual experiences without contribution from the primary visual cortex.

An unexpected result of this study was that when actually viewing colored surfaces, synesthetes do not activate their left V4, the area for color. Right V4 did function similarly for both synesthetes and controls. Ordinarily, viewing colors activates both right and left V4, as well as the early visual areas V1 and V2. The implication, therefore, is that the participation of left V4 in synesthetic color experience renders it unavailable for ordinary color perception— in other words, synesthesia appears to have hijacked an existing brain function. This surprise is consistent with the observation that nonsynesthetes merely imagining colors (compared to performing a visual control task not involving color) do not activate V4. Thus, the brain basis of synesthetic color experience is consistent with real color perception rather than color imagery. This refutes earlier criticisms that synesthetes are just “making it up” or have “overactive imaginations.”

Lastly, this study has largely overthrown the only strong alternative explanation of synesthesia, namely, that it results from childhood learning through association. This claim said that playing with refrigerator magnets or coloring books, for example, makes some children form enduring associations such as “ ‘A’ is red.” Rigorous efforts to train controls to imagine colors in response to words demonstrate that this is not so. Despite training until controls achieved 100 percent accuracy, they showed no activity whatsoever in V4 on either side. To further show that synesthetes did not possess extraordinary associative skills, synesthetes who had claimed no spontaneous color response to music were trained to associate colors with a melody, as were controls; neither group had activity in the V4 region that had activated when synesthetes heard spoken words. Thus, not only was learning ruled out as an explanation, but also the patterns of brain activity could easily distinguish the subjective states that synesthetes claimed to experience (word-color) or denied having (music-color). Taken together, these results support the existence of a direct neural projection from auditory speech areas to the visual color area known as V4.

Those of us who study synesthesia mostly concur now that inheriting a genetic mutation results in a failure in synesthetes’ brains to prune the projections between brain structures that normally exist temporarily during the development of all brains. This is what we call the “neonatal hypothesis” for synesthesia: Everyone is born synesthetic, only to lose the capacity as the brain matures. Because it is not possible to directly map hardwiring in living humans, we are at present debating precisely where these projections might lie, and dreaming up ways to conﬁrm or disprove our conjectures.

The objectivists have ﬁnally gotten a machine proof of synesthesia, but it has disappointed their expectations.

So, the objectivists have ﬁnally gotten a machine proof of synesthesia, but it has disappointed their expectations.

CONVENTION UNDER THREAT

The existence of any physical projection as a basis for synesthesia threatens one of contemporary neuroscience’s widely held concepts, modularity. As initially proposed by Rutgers University philosopher Jerry Fodor, the mind is constructed of independent subsystems that receive inputs only from a speciﬁc category of stimulus and that operate uninﬂuenced by activity in other modules or systems. The concept of modularity originally referred to cognitive domains, but over time has extended into the physical organization of the brain, such that relatively self-contained entities such as V1, V4, and the grapheme area are also referred to as modules. The mental and physical concepts are not wholly comparable, but this is not central to my point. Synesthesia obviously raises the question of whether the concept of modularity per se remains entirely valid.

Another endangered favorite of philosophers and cognitive scientists is functionalism. This concept relates to what is called the “hard problem of consciousness,” namely, the subjective aspect of perception. Functionalism describes the relations among sensory inputs and their neural transformations, the resulting behavior, and our conscious experience. The concept has engendered many varieties of philosophical argument. One popular formulation states that each subjective experience (“quale,” plural “qualia”) is identical to the function with which it is associated. That is, functionalism replaces any supposition that red “feels like” a certain state with, instead, an observable behavior, such as a person saying “red” or pointing to it. Functionalism says that qualia are the functions (inputprocessing-behavioral output) by which they are supported and nothing more.

If so, then two conditions incompatible with functionalism would be two qualia produced by a single function, or two functions producing the same quale. In synesthetes, the quale of “red,” for example, can arise either by optical or nonoptical routes. This is an example of the second condition, since two different neural processes on opposite sides of the brain, one optical and the other synesthetic, are both subjectively experienced as the color red.

Another argument put forth for functionalism is that functions giving rise to qualia must beneﬁt the organism, because evolution selects for traits favoring survival. If this is correct, one should not encounter qualia that interfere with the functions of which they are part. I have already mentioned the situations where a perceptual mismatch slows performance, however, and I give many examples of sensory interference in my textbook, to say nothing of the unpleasant and sometimes disruptive affect accompanying some synesthesiae. Nor is there any positive evidence that the quale of color helps aural or visual word perception. These observations are incompatible with the evolutionary claim of functionalism.

Two different neural processes on opposite sides of the brain, one optical and the other synesthetic, are both subjectively experienced as the color red.

In 1997, Jeffrey Gray was the ﬁrst to notice the danger that synesthesia posed to the hard question of consciousness, and he has studied this problem in depth. Because functionalism purports to be a general account of consciousness, a single negative instance that it cannot explain is sufﬁcient to render it invalid, just as the axiom “All swans are white” can be invalidated by observing a single black swan. If functionalism does not work in synesthesia, it does not work anywhere and thus cannot be a general account of consciousness.

“I watched the black background become pierced by a bright red color that began to form in the middle of the rich velvet blackness. The red began as a small dot of color and grew quite large rather quickly, chasing much of the blackness away. I saw green shapes appear in the midst of the red color and move around the red and black fields.” Carol Steen

The ready objections that synesthetes are not really seeing red—that they are merely being artistic or metaphorical, or saying what they do only because of a vivid memory of some past association such as refrigerator magnets—have already been addressed. Because it is unlikely that philosophers will now succeed in eliminating synesthesia, they must either eliminate functionalism or reﬁne it. I feel conﬁdent they will choose the latter, because philosophers never tire of arguing.

Lastly, synesthesia deals a blow to the staunchest objectivists by showing clearly how perception is not passive, how it is not an impression in the brain transferred by objective physics in the world “out there” (philosophers call this direct realism). When a synesthete responds to the word “butter” by saying “blue circles moving off to the right,” she demonstrates a lack of correspondence, let alone an identity, between the physical world “out there” that produces the percept and the percept itself. Many other approaches have supported this notion that perception is active and constructive; synesthesia happily provides a clear example.

So much for the wrecking ball. What issues might synesthesia illuminate? Two big ones are the so-called binding problem and metaphor.

THE BINDING PROBLEM

Diverse perceptual attributes (such as color or shape) are processed in different areas of my brain, yet I perceive an apple as a unitary entity, not something red + round + edible. What is more, attributes are processed not only in different locations but also at different times in my brain. For example, color is perceived before motion, which is perceived before form. How all of these sundry, asynchronous attributes get bound into a seamless perception—red apple—endlessly bafﬂes neuroscientists. Inasmuch as synesthesia binds perceptual qualia together in anomalous combinations, might it not say something useful about the process of binding in general?

There is a further twist. I mentioned earlier that synesthesia’s most common manifestation is a coupling of sensory qualia to categories of knowledge: for example, color, ﬂavor, texture, locus, or conﬁguration may be bound to letters and integers, members of a serially ordered set (such as days of the week), words, or even symbols such as braille. Consider how many neurological syndromes (the agnosias) as well as imaging studies demonstrate that we think in categories. In prosopagnosia, for example, stricken individuals can no longer recognize faces. They recognize a face as a face, but cannot say whose face it is. Their larger failure is in comprehending examples within a category. Thus, despite all their previous knowledge and skill, a stricken bird watcher says that all the birds look alike, a farmer can no longer distinguish his cows, and a gardener cannot tell one plant from another.

An enduring puzzle of neuroscience is how, out of a constantly changing and inﬁnite energy ﬂux, the brain— whose resources are finite—assigns objects their constant features.

Might synesthesia relate to the brain’s search for constancy and the assignment of essential features that constitute a category? An enduring puzzle of neuroscience is how, out of a constantly changing and inﬁnite energy ﬂux, the brain—whose resources are ﬁnite—assigns objects their constant features.

Color and form, so prominent in synesthesia, are properties constructed by the brain through what are called constancy operations. For example, most of us accept the explanation that something looks red because it reﬂects red wavelengths more than others, but color is actually a property of brains and not of the physical world. For surface colors to be perceived as constant despite ever-changing illumination, it is precisely the wavelength composition of reﬂected light that the brain must ignore. Grass looks green, whether it is in bright sunlight or shade, despite large differences in wavelength composition of the light. Similarly, all constructed properties require that the brain discount certain things. With color, it is wavelength composition of reﬂected light that the brain must ignore; with form, it is the viewing angle; and with size, it is viewing distance.

Synesthesia has led me over time to favor a model of brain organization called the distributed system. The prime features of this model are a distribution of function (hence the name) across structures—as in neural networks—and simultaneity of activity on several levels, compared to the older and more familiar hierarchical and sequential cascade, in which a module is assumed to complete its transformation of neural inputs before passing the result on to the next module in the sequence. This older idea may be likened to stations in a factory connected by a conveyor belt by means of which one thing after another is added, whereas the distributed system is like different authors simultaneously writing separate chapters of a book without fully knowing how the other chapters end. The distributed system also departs from the older idea of a strict one-to-one mapping of function to anatomy, depending instead on topological relations and convergent-divergent connections among brain modules. These two features result in the multiple mapping of a given function, as seen in the numerous modules pertaining to vision, some of which we understand better than others (such as V4 for color, or V5 for motion and direction). Relevant to synesthesia, what are called transmodal modules (meaning “not pertaining to any single sense”) do three things: They construct multisensory representations of the world, they provide memory and affect to experience, and they critically participate in establishing categories via groups of coarsely tuned neurons.

The answer to synesthesia will not be a “where” but a “what.”

This model organizes brain tissue into ﬁve major networks and many lesser distributed systems. In any one such system, a given cerebral module participates in more than one cognitive function and connects with several-to-many other nodes. A given function is not so much localized in the sense of classical neurology, but exists as the dominant process within its distributed system at any given time. Multiple synaptic levels are active simultaneously, each node inﬂuencing the state of adjacent levels (as in the example of our simultaneous authors). Such organization reminds us that localization is a function of probability—and not just in this model but in any scheme of neural organization. (Try drawing the boundaries of Wernicke’s area on a standard brain atlas—you can’t.) Scans mislead us by emphasizing peak probabilities, which we misconstrue as ﬁxedly anatomical. The answer to synesthesia will not be a “where” but a “what.”

It would thus be wrong for me to leave the impression that V4 is the seat of synesthesia: Any module found active by a scan (or other means) is really just one node in the distributed system underlying expression. The totality of synesthetic experience involves more than the conscious perception of a single quale, as I hope I have conveyed throughout this article. My comments regarding the participation of transmodal modules in synesthesia are not incompatible with the idea, mentioned earlier, that an inherited genetic mutation causes extraordinary, one-way projections between cerebral modules that underlie very speciﬁc functions. A connection between, say, the grapheme area that allows one to understand written numbers and the V4 color area does not fully “explain” synesthetically colored numbers, however, because it leaves out the affect of the experience, its memorability, whether the synesthetic color moves, has a given spatial location, and so forth. As V. S. Ramachandran points out, what are called transcription factors can partly solve this shortcoming by causing the gene’s effects to be expressed either discretely or diffusely—or anywhere between—in the brain. Such variability goes a long way toward explaining the observed variety of synesthetic experience, and why some people have only one kind whereas others have three or four different kinds of synesthesia. Thus, transcription factors expressed in different places through-out the brain could account, theoretically at least, for subsidiary features of synesthesia such as memorability and affective charge. But it is precisely this necessity of widespread expression that makes me point out why synesthetic experience per se cannot be localized to any one physical spot in the brain and why scans mislead us in this regard.

I proposed a cognitive continuum extending from perception to synesthesia to metaphor to language. With time, others have come to concur.

METAPHOR AND LANGUAGE

The heterogeneity of the synesthetic experience connotes more than wide variety of perceptual combinations. There is also heterogeneity in the depth of subjective experience, from purely sensory-sensory, to categorical-sensory, to verbal-sensory. In this last, even a concept—just thinking of the number 5, say, or a person named Marion—is sufﬁcient to trigger synesthesia. Some time ago, both Lawrence Marks and I proposed a cognitive continuum extending from perception to synesthesia to metaphor to language. With time, others have come to concur.

Systematic correspondences exist among dimensions of a given sense for synesthetes and nonsynesthetes alike. For example, both say that louder tones are brighter than soft tones, that higher ones are smaller than lower ones, and that low tones are both larger and darker than high ones. The perceptual similarities that yield such orderly relationships among pitch, loudness, brightness, and size, for example, turn out to be rooted in fundamental similarities of physical experience itself. Perceptual similarities, synesthetic equivalences, and metaphoric identities in turn become available to the more abstract knowledge that is embodied in language. In other words, the acquisition of metaphor relies not on a capacity for verbal abstraction, as many mistakenly believe, but on our physical interaction with the world. The subjective-objective dichotomy of experience should be turned into a unity, because we need both points of view.

Objectivity fails to see how the human system of concepts is metaphoric, involving an imaginative understanding of one thing in terms of another. We elaborate the metaphor “The mind is an entity” into another metaphor, “The mind is a machine,” when we say, “He ran out of steam.” Metaphors emphasize some aspects of an object but hide others. The machine metaphor paints the mind as having a source of power, an on-off state, and an expected level of efﬁciency, but it hides the vagaries of thought, its ability to make sense of fragmentary information, and the unexpected suddenness of insight. By switching metaphors, we alter how we comprehend a thing.

Synesthetes who see colored letters have their own individual alphabets. Carol Steen, who painted this representation of her alphabet, says she saw many of the more brightly colored letters as a young child, but the iridescent and metallic colored letters did not appear until she was in her 30s. Many synesthetes say that their perceptions become richer and more complex as they age.

I Carol Steen

Subjectivity fails to see that even the most imaginative flights occur in a context of objective experience gained by living in a physical and cultural world. Increasingly, science is viewing metaphor as an emergent property of mind that is rooted in the body. As semiotics have long known, meaning inheres in affect, which the body feels as physical and the mind apprehends as mental. Because metaphor perceives the similar in the dissimilar, it also points to constancy and categorization, features germane to synesthesia. Perhaps a tendency to map one concept to another unconventionally even underlies what appears to be synesthetes’ distinctive approach to creativity.

One implication of a continuum from perception to synesthesia to metaphor to language is that synesthesia resides universally in each of us but, for reasons yet unknown, rises to consciousness in only a few. Heinz Werner suggested as much in the 1930s but technology takes time to catch up with ideas. Two bits of recent work support this conjecture. One study found that synesthesia is 100 times more frequent during Zen meditation; the other conﬁrmed the ability of both blind and sighted persons to “see” video impulses fed into an electrode array placed on the tongue. We do not see with our eyes, anyway, but with our brains. What this latter demonstration shows is that tactile sensations on the tongue can be unconventionally bound to discern form, movement, direction, spatial location, and other qualia that we conventionally ascribe to vision. The capacity for anomalous binding, which is the essence of synesthesia, is therefore latent in all brains.

Nature reveals herself through exceptions. Those objectivists who tried to dismiss synesthesia throughout its history seem to have forgotten this maxim.

Nature reveals herself through exceptions. Those objectivists who tried to dismiss synesthesia throughout its history seem to have forgotten this maxim. Far from being a mere curiosity irrelevant to real questions, synesthesia turns out to illuminate a wide swath of mental life and forces us to rethink some fundamental issues regarding mind and brain. At present, I can think of nothing more relevant to our quest for self-understanding.

References

Bogen, JE, Bogen, GM. “Wernicke’s region–Where is it?” Annals of the New York Academy of Science 1975; 280: 834-843.

Cytowic, RE. The Man Who Tasted Shapes. Cambridge. MIT Press, 1998.

Cytowic, RE. Synesthesia: A Union of The Senses, 2d ed. Cambridge. MIT Press, 2002.

Gray, JA, et al. “Possible questions of synesthesia for the hard question of consciousness.” In S Baron-Cohen & JE Harrison, eds., Synaesthesia: Classic and Contemporary Readings. Oxford. Blackwell, 1997: 173-181.

Luria, AR. The Mind of a Mnemonist. New York. Basic Books, 1968.

Marks, LE. The Unity of The Senses: Interrelations Among the Modalities. New York. Academic Press, 1978.

Nunn, JA, et al. “Functional magnetic resonance imaging of synesthesia: activation of V4/V8 by spoken words.” Nature Neuroscience 2002; 5(4): 571-575.

Paulesu, E, et al. “The physiology of coloured-hearing: a PET activation study of colour-word synaesthesia.” Brain 1995; 118: 661-676.

Ramachandran, VS, Hubbard EM. “Synesthesia: a window into perception, thought, and language.” Journal of Consciousness Studies 2001; 8(12): 3-34.

Zeki, S. A Vision of the Brain. Oxford. Blackwell, 1993.

Wednesday, January 7, 2015

Synesthesia