How Does the Phenomenology of Conscious Experience Form?
How does consciousness arise? If the phenomena of subjective reality (thoughts, images) do not possess physical properties, what is their relationship to the workings of the brain and the body as a whole? How can we explain their causal influence on processes in our bodies that are reflected in our behavior (In other words, how can the immaterial affect the material? — Ed.)? Is there any causal connection at all? These questions are known in science as the “hard problem of consciousness,” classically formulated by David Chalmers:
“Some living beings are undoubtedly subjects of experience. But the question of how these organisms become subjects of experience is perplexing. Why is it that when our cognitive systems process visual and auditory information, we have visual or auditory impressions, perceiving the quality of color (a deep blue) or sound (middle C)? How can we explain the emergence of mental images or the experience of emotions? It is generally accepted that experience arises on a physical basis, but we have no clear explanation of why and how it arises from it. Why should physical processing give rise to rich inner experiences? There is no objective logic to this, yet this is exactly what happens.”
Neuroscientist Anil Seth, among others working on these questions today, acknowledges that it is currently impossible to solve the “hard problem of consciousness.” In his book Being You (Alpina Non-Fiction), he breaks it down into components and examines three aspects of what makes up our being: levels of consciousness (ranging from a complete absence of conscious experience, as in coma or brain death, to vivid states of awareness during wakefulness), its content based on perception (sounds, visual images, smells, emotions, thoughts, and beliefs that make up our inner world), and the process of forming the sense of “self”.
When it comes to the content of consciousness, to understand how and from what it is formed, Seth suggests the following thought experiment:
“Try to really put yourself in its [the brain’s] place and imagine what it’s like to be locked in the bony prison of the skull, trying to figure out what’s happening outside, in the external world. No light, no sounds, nothing gets in—just total darkness and silence. The only thing available to form impressions is an endless stream of electrical signals bombarding you, which are only indirectly related to whatever is happening outside. These signals from the senses don’t come with labels like ‘this is from a coffee cup’ or ‘this is from a tree.’ They don’t even indicate their modality—whether they’re visual, auditory, tactile, or from a lesser-known sense like thermoreception (feeling heat and cold) or proprioception (sensing the body’s position in space).”
Reflecting on how the brain transforms these ambiguous sensory signals into a coherent perceptual world filled with subjects and objects, Anil Seth proposes a kind of Copernican revolution in our understanding of perception: in his view, perceptual experience is determined by the content of top-down predictions, not bottom-up sensory signals. In other words, we never “experience” the sensory signals themselves; we only experience their interpretation.
“We tend to think that our senses reveal the outside world to our conscious mind as it really is. With this mindset, it’s natural to see perception as a bottom-up process of feature recognition—’reading’ the reality around us. In fact, we perceive a top-down, inside-out neural fantasy that reality merely constrains, rather than looking at reality through a transparent window.”
As an example, he discusses visual perception. How does the phenomenology—the subjective experience—of color, say red, arise? Not only does our visual system respond to only a small segment of the electromagnetic spectrum, giving us an inherently limited view of the world, but the color we perceive is determined by a complex interaction of environmental conditions, the light reflecting off surfaces, and the brain’s probable assumptions about how this interaction unfolds.
Below is an excerpt in which, using several perceptual illusions, he explores how we continuously build a probable model of the world and correct prediction errors every microsecond, why perception is a creative, generative act based on the unique features of our personal biography and biology, how the brain is a “prediction machine,” and how the sum of perceptual experience is a “neural fantasy constrained by a constant stream of the most probable perceptual assumptions—controlled hallucinations.”
Perception from the Inside Out (Excerpt)
Let’s look at how perceptual expectations shape conscious experience—here are three examples you can try for yourself.
The Blue and Gold Dress Phenomenon
If you were on social media or reading the news in February 2015, you probably remember the frenzy over the “blue or white dress” phenomenon. One Wednesday morning, I was bombarded with emails and voice messages at work. I had recently co-authored a book on visual illusions, and the media were scrambling for explanations of this viral internet event. The “phenomenon” was a photo of a dress that some people saw as blue and black, while others saw it as white and gold. Those who saw one color combination were so sure of their perception that they couldn’t imagine how anyone could see it differently, sparking heated online debates.
At first, I thought it was a prank. Like the first four lab colleagues I showed the picture to, I clearly saw a black and blue dress, so when the fifth person said “white and gold,” I was stunned and intrigued. The lab, like the rest of the world, split roughly in half: some saw the dress as black and blue, others as white and gold.
An hour later, I was on the BBC trying to explain what was going on. Everyone gradually agreed that the effect was due to adjustments for the light source—that is, the influence of overall lighting on color perception. The idea was that this process could differ from person to person, usually without us noticing, and only now did it play a decisive role.
People quickly noticed that the photo was overexposed and lacked context (the dress took up most of the image), which could easily fool the brain as it tried to use context for its conclusions. If your visual system is used to yellowish artificial light—say, you spend a lot of time indoors—you’re likely to interpret the dress as blue and black, assuming the light source is yellowish. Conversely, if you’re more accustomed to sunlight and bluish natural light, you might see the dress as white and gold.
Everyone started experimenting—staring at the photo in a dim room, then running outside into daylight; comparing the prevalence of white-and-gold viewers to the average number of sunny days in different countries; checking if more older people saw blue and black than younger ones. A whole cottage industry of amateur hypotheses sprang up overnight.
The fact that the same image could produce such different sensory experiences, and that people were so confident in their own perception, convincingly shows that our perception of the world is an internal construct shaped by the unique features of our personal biology and biography. We usually assume everyone sees the world more or less the same way, and in most cases, that’s probably true. But even if it is, it’s not because red chairs are truly red, but because only very unusual circumstances—like the dress photo—can reveal the subtle differences in how our brains select the most probable assumptions.
Adelson’s Checker-Shadow Illusion
As a second example, let’s look at the popular optical illusion known as “Adelson’s Checker-Shadow Illusion.” It shows that predictions affect perception not just in unusual situations like the dress, but constantly, at every turn. Look at the checkerboard on the left side of Figure 5 and compare squares A and B. Most likely, A will appear darker than B. That’s how I see it, and so does everyone I’ve shown it to. There’s no hint of individual differences here.
But in reality, A and B are exactly the same shade of gray. This is proven by the checkerboard on the right side of the figure, where A and B are connected by two strips of identical gray. If you look closely at the resulting rectangle, you’ll see the shades don’t change—there are no color transitions. A and B are truly the same gray, even though on the left checkerboard they still look different. Knowing they’re the same color doesn’t help. I’ve looked at these images thousands of times, and A and B on the left still refuse to look identical (Perception that isn’t affected by knowledge is called “cognitively impenetrable.” — Author’s note).
The reason is that the perception of gray is determined not by the light waves coming from A and B (which are the same), but by the brain’s most probable assumption about what causes that particular combination of wavelengths, and this assumption, as with the dress, depends on context. Square B is in shadow, square A is not, and the brain’s visual networks are hardwired to expect that objects in shadow appear darker. Just as the brain adjusts its perceptual conclusions based on overall lighting, it adjusts its conclusions about the shade of square B based on prior knowledge about shadows. That’s why on the left checkerboard, B seems lighter than A, which isn’t in shadow. On the right checkerboard, the shadow context is disrupted by the gray strips, and we see that A and B are actually identical.
This happens completely automatically. You’re not aware (at least until now) that your brain uses built-in expectations about shadows when making perceptual predictions. And this isn’t a flaw in our visual system. An effective visual system isn’t a light meter like photographers use. The function of perception, at least at first approximation, is to calculate the most probable sources of sensory signals, not to make us aware of the signals themselves, whatever that awareness might be.
The Mooney Image Illusion
The last example shows how quickly new predictions can influence conscious perception. Look at the image below (Figure 6). Most likely, you’ll just see a jumble of black and white blobs. Then, after you finish this sentence, turn a few pages and look at the photo there (Figure 7).
Have you looked? Great. Now look back at Figure 6—this time, the image will seem different. Where there was once a mess of spots, now you’ll see objects, clear outlines, and a scene. This is a black-and-white illusion, or “Mooney image.” Once you see the subject of such a picture, you’ll always see it. To create a Mooney image, a photo is converted to grayscale and then carefully segmented using a thresholding method, which removes all fine details, leaving only stark black and white. If done properly on the right image, it’s very hard to tell what’s in the picture until you see the original, after which the black-and-white mess suddenly becomes a coherent scene.
This example is notable because, as you look at the black-and-white jumble, your eyes are receiving the same sensory stimuli as the first time—the stimuli themselves haven’t changed at all. Only your brain’s predictions about the sources of those sensory data have changed, and it’s those predictions that alter your conscious perception.
This phenomenon isn’t limited to vision. A compelling example in hearing is “sine-wave speech”—a processed spoken phrase where all the high frequencies that help us recognize speech are removed. The result usually sounds like a noisy whistle, in which nothing can be made out—a kind of auditory equivalent of the black-and-white illusion. Then you listen to the original, unprocessed speech, and then the sine-wave version again—and suddenly everything becomes clear. Just like with Mooney images, a clear prediction about the source of sensory signals enriches perceptual experience.
Perception as a Creative, Generative Act
Taken together, these examples (admittedly, intentionally simple) show that perception is a creative, generative act—a proactive, context-dependent interpretation of sensory signals and interaction with them. As I’ve said, the principle that perceptual experience is built on predictions generated in the brain applies everywhere—not just to vision and hearing, but to all our perception, all the time.
This means we never perceive the world as it “really is.” Even Kant, with his concept of the noumenon, noted that it’s hard to imagine what such perception would be like. We’ve already seen that even something as simple as color exists only in the interaction between the world and the mind. So, as surprising as it may be that perceptual illusions—like those we’ve discussed—reveal the difference between what we see (hear, feel) and what’s actually there, we should try not to judge perceptual experience solely by how “accurately” it matches reality. In this sense, accurate (“veridical”) perception is a chimera. The controlled hallucination of the world given to us in sensation was shaped by evolution to increase our chances of survival, not to serve as a transparent window onto external reality, which has no conceptual meaning. We’ll explore these ideas further in later chapters, but first, it’s worth addressing some counterarguments.
Figures 6 and 7 are from Teufel, C., Dakin, S. C. & Fletcher, P. C. (2018), ‘Prior object-knowledge sharpens properties of early visual feature detectors’, Scientific Reports, 8: 01853. Reprinted with permission from the authors and under the Creative Commons Attribution 4.1 International License, with thanks to Christoph Teufel.
Counterarguments
The first is that the theory of perception as controlled hallucination denies the existence of real-world absolutes. “If everything we sense is just a hallucination,” you might object, “go jump in front of a train and see what happens.”
Nothing I’ve said should be taken as denying the existence of objects and phenomena in the world around us, whether it’s an oncoming train, a cat, or a cup of coffee. The word “controlled” in our term is just as important as “hallucination.” By describing perception this way, we don’t mean it’s made up or can be anything at all; we mean that the representation of reality in perceptual experience is a construct of our brain.
It’s helpful here to understand the difference between “primary” and “secondary” qualities, as Enlightenment philosopher John Locke called them. He considered primary qualities to be those that exist independently of the observer, such as extension, density, and motion. A speeding train has these primary qualities in abundance, so you’d better not jump in front of it, regardless of whether you observe it or what your views on perception are. Secondary qualities, on the other hand, are those that depend on the observer. These are properties of objects that cause sensations—or “ideas”—in the mind, and they can’t be said to exist apart from the object. A good example of a secondary quality is color, since color perception depends on the interaction of a particular perceptual apparatus with the object.
From the perspective of controlled hallucination, both primary and secondary qualities of objects can give rise to perceptual experience through an active, constructive process. However, in neither case will the content of that perceptual experience be identical to the corresponding property of the object.
The second counterargument concerns our ability to perceive new things. It might seem that for anything we ever perceive, a most probable assumption must already be formed, and so we’re forever locked in a perceptual world of prior expectations. Imagine you’ve never seen a gorilla—not in real life, not on TV, not in movies, not even in books—and suddenly one comes walking toward you down the street. I guarantee you’d still see the gorilla, and it would be a new and probably quite frightening perceptual experience. But how is this possible in a world of prior expectations? In short, the experience of “seeing a gorilla” isn’t entirely new. A gorilla is an animal with arms, legs, and fur, and you or your ancestors have certainly seen other creatures with similar or partially similar features. More generally, a gorilla is an object with distinct (albeit hairy) outlines, it moves in a reasonably predictable way, and it reflects light like other objects of similar size, color, and texture. The novelty of the “seeing a gorilla” experience is built on perceptual predictions operating at many different levels of detail and acquired over different periods of time—from predictions about lighting and outlines to predictions about faces and poses—all of which combine to form new, general, most probable perceptual assumptions, and so you see the gorilla for the first time.
To answer in more detail, we need to learn more about how the brain performs its incredibly complex neural acrobatics involved in perceptual inference, which we’ll explore in the next chapter.