How to Create a Mind: The Secret of Human Thought Revealed

A Hierarchy of Patterns

I have repeated the simple experiments and observations described in the previous chapter thousands of times in myriad contexts. The conclusions from these observations necessarily constrain my explanation for what the brain must be doing, just as the simple experiments on time, space, and mass that were conducted in the early and late nineteenth century necessarily constrained the young Master Einstein’s reflections on how the universe functioned. In the discussion that follows I’ll also factor in some very basic observations from neuroscience, attempting to avoid the many details that are still in contention.

First, let me explain why this section specifically discusses the neocortex (from the Latin meaning “new rind”). We do know the neocortex is responsible for our ability to deal with patterns of information and to do so in a hierarchical fashion. Animals without a neocortex (basically nonmammals) are largely incapable of understanding hierarchies.¹ Understanding and leveraging the innately hierarchical nature of reality is a uniquely mammalian trait and results from mammals’ unique possession of this evolutionarily recent brain structure. The neocortex is responsible for sensory perception, recognition of everything from visual objects to abstract concepts, controlling movement, reasoning from spatial orientation to rational thought, and language—basically, what we regard as “thinking.”

The human neocortex, the outermost layer of the brain, is a thin, essentially two-dimensional structure with a thickness of about 2.5 millimeters (about a tenth of an inch). In rodents, it is about the size of a postage stamp and is smooth. An evolutionary innovation in primates is that it became intricately folded over the top of the rest of the brain with deep ridges, grooves, and wrinkles to increase its surface area. Due to its elaborate folding, the neocortex constitutes the bulk of the human brain, accounting for 80 percent of its weight. Homo sapiens developed a large forehead to allow for an even larger neocortex; in particular we have a frontal lobe where we deal with the more abstract patterns associated with high-level concepts.

This thin structure is basically made up of six layers, numbered I (the outermost layer) to VI. The axons emerging from the neurons in layers II and III project to other parts of the neocortex. The axons (output connections) from layers V and VI are connected primarily outside of the neocortex to the thalamus, brain stem, and spinal cord. The neurons in layer IV receive synaptic (input) connections from neurons that are outside the neocortex, especially in the thalamus. The number of layers varies slightly from region to region. Layer IV is very thin in the motor cortex, because in that area it largely does not receive input from the thalamus, brain stem, or spinal cord. Conversely, in the occipital lobe (the part of the neocortex usually responsible for visual processing), there are three additional sublayers that can be seen in layer IV, due to the considerable input flowing into this region, including from the thalamus.

A critically important observation about the neocortex is the extraordinary uniformity of its fundamental structure. This was first noticed by American neuroscientist Vernon Mountcastle (born in 1918). In 1957 Mountcastle discovered the columnar organization of the neocortex. In 1978 he made an observation that is as significant to neuroscience as the Michelson-Morley ether-disproving experiment of 1887 were to physics. That year he described the remarkably unvarying organization of the neocortex, hypothesizing that it was composed of a single mechanism that was repeated over and over again,² and proposing the cortical column as that basic unit. The differences in the height of certain layers in different regions noted above are simply differences in the amount of interconnectivity that the regions are responsible for dealing with.

Mountcastle hypothesized the existence of mini-columns within columns, but this theory became controversial because there were no visible demarcations of such smaller structures. However, extensive experimentation has revealed that there are in fact repeating units within the neuron fabric of each column. It is my contention that the basic unit is a pattern recognizer and that this constitutes the fundamental component of the neocortex. In contrast to Mountcastle’s notion of a mini-column, there is no specific physical boundary to these recognizers, as they are placed closely one to the next in an interwoven fashion, so the cortical column is simply an aggregate of a large number of them. These recognizers are capable of wiring themselves to one another throughout the course of a lifetime, so the elaborate connectivity (between modules) that we see in the neocortex is not prespecified by the genetic code, but rather is created to reflect the patterns we actually learn over time. I will describe this thesis in more detail, but I maintain that this is how the neocortex must be organized.

It should be noted, before we further consider the structure of the neocortex, that it is important to model systems at the right level. Although chemistry is theoretically based on physics and could be derived entirely from physics, this would be unwieldy and infeasible in practice, so chemistry has established its own rules and models. Similarly, we should be able to deduce the laws of thermodynamics from physics, but once we have a sufficient number of particles to call them a gas rather than simply a bunch of particles, solving equations for the physics of each particle interaction becomes hopeless, whereas the laws of thermodynamics work quite well. Biology likewise has its own rules and models. A single pancreatic islet cell is enormously complicated, especially if we model it at the level of molecules; modeling what a pancreas actually does in terms of regulating levels of insulin and digestive enzymes is considerably less complex.

The same principle applies to the levels of modeling and understanding in the brain. It is certainly a useful and necessary part of reverse-engineering the brain to model its interactions at the molecular level, but the goal of the effort here is essentially to refine our model to account for how the brain processes information to produce cognitive meaning.

American scientist Herbert A. Simon (1916–2001), who is credited with cofounding the field of artificial intelligence, wrote eloquently about the issue of understanding complex systems at the right level of abstraction. In describing an AI program he had devised called EPAM (elementary perceiver and memorizer), he wrote in 1973, “Suppose you decided that you wanted to understand the mysterious EPAM program that I have. I could provide you with two versions of it. One would be…the form in which it was actually written—with its whole structure of routines and subroutines…. Alternatively, I could provide you with a machine-language version of EPAM after the whole translation had been carried out—after it had been flattened so to speak…. I don’t think I need argue at length which of these two versions would provide the most parsimonious, the most meaningful, the most lawful description…. I will not even propose to you the third…of providing you with neither program, but instead with the electromagnetic equations and boundary conditions that the computer, viewed as a physical system, would have to obey while behaving as EPAM. That would be the acme of reduction and incomprehensibility.”³

There are about a half million cortical columns in a human neocortex, each occupying a space about two millimeters high and a half millimeter wide and containing about 60,000 neurons (resulting in a total of about 30 billion neurons in the neocortex). A rough estimate is that each pattern recognizer within a cortical column contains about 100 neurons, so there are on the order of 300 million pattern recognizers in total in the neocortex.

As we consider how these pattern recognizers work, let me begin by saying that it is difficult to know precisely where to begin. Everything happens simultaneously in the neocortex, so there is no beginning and no end to its processes. I will frequently need to refer to phenomena that I have not yet explained but plan to come back to, so please bear with these forward references.

Human beings have only a weak ability to process logic, but a very deep core capability of recognizing patterns. To do logical thinking, we need to use the neocortex, which is basically a large pattern recognizer. It is not an ideal mechanism for performing logical transformations, but it is the only facility we have for the job. Compare, for example, how a human plays chess to how a typical computer chess program works. Deep Blue, the computer that defeated Garry Kasparov, the human world chess champion, in 1997 was capable of analyzing the logical implications of 200 million board positions (representing different move-countermove sequences) every second. (That can now be done, by the way, on a few personal computers.) Kasparov was asked how many positions he could analyze each second, and he said it was less than one. How is it, then, that he was able to hold up to Deep Blue at all? The answer is the very strong ability humans have to recognize patterns. However, we need to train this facility, which is why not everyone can play master chess.

Kasparov had learned about 100,000 board positions. That’s a real number—we have established that a human master in a particular field has mastered about 100,000 chunks of knowledge. Shakespeare composed his plays with 100,000 word senses (employing about 29,000 distinct words, but using most of them in multiple ways). Medical expert systems that have been built to represent the knowledge of a human medical physician have shown that a typical human medical specialist has mastered about 100,000 concepts in his or her domain. Recognizing a chunk of knowledge from this store is not straightforward, as a particular item will present itself a little bit differently each time it is experienced.

Armed with his knowledge, Kasparov looks at the chessboard and compares the patterns that he sees to all 100,000 board situations that he has mastered, and he does all 100,000 comparisons simultaneously. There is consensus on this point: All of our neurons are processing—considering the patterns—at the same time. That does not mean that they are all firing simultaneously (we would probably fall to the floor if that happened), but while doing their processing are considering the possibility of firing.

How many patterns can the neocortex store? We need to factor in the phenomenon of redundancy. The face of a loved one, for example, is not stored once but on the order of thousands of times. Some of these repetitions are largely the same image of the face, whereas most show different perspectives of it, different lighting, different expressions, and so on. None of these repeated patterns are stored as images per se (that is, as two-dimensional arrays of pixels). Rather, they are stored as lists of features where the constituent elements of a pattern are themselves patterns. We’ll describe below more precisely what these hierarchies of features look like and how they are organized.

If we take the core knowledge of an expert as consisting of about 100,000 “chunks” of knowledge (that is, patterns) with a redundancy estimate of about 100 to 1, that gives us a requirement of 10 million patterns. This core expert knowledge is built on more general and extensive professional knowledge, so we can increase the order of magnitude of patterns to about 30 to 50 million. Our everyday “commonsense” knowledge as a human being is even greater; “street smarts” actually require substantially more of our neocortex than “book smarts.” Including this brings our estimate to well over 100 million patterns, taking into account the redundancy factor of about 100. Note that the redundancy factor is far from fixed—very common patterns will have a redundancy factor well into the thousands, whereas a brand-new phenomenon may have a redundancy factor of less than 10.

As I will discuss below, our procedures and actions also comprise patterns and are likewise stored in regions of the cortex, so my estimate of the total capacity of the human neocortex is on the order of low hundreds of millions of patterns. This rough tally correlates well with the number of pattern recognizers that I estimated above at about 300 million, so it is a reasonable conclusion that the function of each neocortical pattern recognizer is to process one iteration (that is, one copy among the multiple redundant copies of most patterns in the neocortex) of a pattern. Our estimates of the number of patterns that a human brain is capable of dealing with (including necessary redundancy) and the number of physical pattern recognizers happen to be the same order of magnitude. It should be noted here that when I refer to “processing” a pattern, I am referring to all of the things we are able to do with a pattern: learn it, predict it (including parts of it), recognize it, and implement it (either by thinking about it further or through a pattern of physical movement).

Three hundred million pattern processors may sound like a large number, and indeed it was sufficient to enable Homo sapiens to develop verbal and written language, all of our tools, and other diverse creations. These inventions have built upon themselves, giving rise to the exponential growth of the information content of technologies as described in my law of accelerating returns. No other species has achieved this. As I discussed, a few other species, such as chimpanzees, do appear to have a rudimentary ability to understand and form language and also to use primitive tools. They do, after all, also have a neocortex, but their abilities are limited due to its smaller size, especially of the frontal lobe. The size of our own neocortex has exceeded a threshold that has enabled our species to build ever more powerful tools, including tools that can now enable us to understand our own intelligence. Ultimately our brains, combined with the technologies they have fostered, will permit us to create a synthetic neocortex that will contain well beyond a mere 300 million pattern processors. Why not a billion? Or a trillion?

The Structure of a Pattern

The pattern recognition theory of mind that I present here is based on the recognition of patterns by pattern recognition modules in the neocortex. These patterns (and the modules) are organized in hierarchies. I discuss below the intellectual roots of this idea, including my own work with hierarchical pattern recognition in the 1980s and 1990s and Jeff Hawkins (born in 1957) and Dileep George’s (born in 1977) model of the neocortex in the early 2000s.

Each pattern (which is recognized by one of the estimated 300 million pattern recognizers in the neocortex) is composed of three parts. Part one is the input, which consists of the lower-level patterns that compose the main pattern. The descriptions for each of these lower-level patterns do not need to be repeated for each higher-level pattern that references them. For example, many of the patterns for words will include the letter “A.” Each of these patterns does not need to repeat the description of the letter “A” but will use the same description. Think of it as being like a Web pointer. There is one Web page (that is, one pattern) for the letter “A,” and all of the Web pages (patterns) for words that include “A” will have a link to the “A” page (to the “A” pattern). Instead of Web links, the neocortex uses actual neural connections. There is an axon from the “A” pattern recognizer that connects to multiple dendrites, one for each word that uses “A.” Keep in mind also the redundancy factor: There is more than one pattern recognizer for the letter “A.” Any of these multiple “A” pattern recognizers can send a signal up to the pattern recognizers that incorporate “A.”

The second part of each pattern is the pattern’s name. In the world of language, this higher-level pattern is simply the word “apple.” Although we directly use our neocortex to understand and process every level of language, most of the patterns it contains are not language patterns per se. In the neocortex the “name” of a pattern is simply the axon that emerges from each pattern processor; when that axon fires, its corresponding pattern has been recognized. The firing of the axon is that pattern recognizer shouting the name of the pattern: “Hey guys, I just saw the written word ‘apple.’”

Three redundant (but somewhat different) patterns for “A” feeding up to higher-level patterns that incorporate “A.”

The third and final part of each pattern is the set of higher-level patterns that it in turn is part of. For the letter “A,” this is all of the words that include “A.” These are, again, like Web links. Each recognized pattern at one level triggers the next level that part of that higher-level pattern is present. In the neocortex, these links are represented by physical dendrites that flow into neurons in each cortical pattern recognizer. Keep in mind that each neuron can receive inputs from multiple dendrites yet produces a single output on an axon. That axon, however, can then in turn transmit to multiple dendrites.

To take some simple examples, the simple patterns on the next page are a small subset of the patterns used to make up printed letters. Note that every level constitutes a pattern. In this case, the shapes are patterns, the letters are patterns, and the words are also patterns. Each of these patterns has a set of inputs, a process of pattern recognition (based on the inputs that take place in the module), and an output (which feeds to the next higher level of pattern recognizer).

Southwest to north-central connection:

Southeast to north-central connection:

Horizontal crossbar:

Leftmost vertical line:

Concave region facing south:

Bottom horizontal line:

Top horizontal line:

Middle horizontal line:

Loop constituting upper region:

The above patterns are constituents of the next higher level of pattern, which is a category called printed letters (there is no such formal category within the neocortex, however; indeed, there are no formal categories).

“A”:

Two different patterns, either of which constitutes “A,” and two different patterns at a higher level (“APPLE” and “PEAR”) of which “A” is a part.

“P”:

Patterns that are part of the higher-level pattern “P.”

“L”:

Patterns that are part of the higher-level pattern “L.”

“E”:

Patterns that are part of the higher-level pattern “E.”

These letter patterns feed up to an even higher-level pattern in a category called words. (The word “words” is our language category for this concept, but the neocortex just treats them only as patterns.)

“APPLE”:

In a different part of the cortex is a comparable hierarchy of pattern recognizers processing actual images of objects (as opposed to printed letters). If you are looking at an actual apple, low-level recognizers will detect curved edges and surface color patterns leading up to a pattern recognizer firing its axon and saying in effect, “Hey guys, I just saw an actual apple.” Yet other pattern recognizers will detect combinations of frequencies of sound leading up to a pattern recognizer in the auditory cortex that might fire its axon indicating, “I just heard the spoken word ‘apple.’”

Keep in mind the redundancy factor—we don’t just have a single pattern recognizer for “apple” in each of its forms (written, spoken, visual). There are likely to be hundreds of such recognizers firing, if not more. The redundancy not only increases the likelihood that you will successfully recognize each instance of an apple but also deals with the variations in real-world apples. For apple objects, there will be pattern recognizers that deal with the many varied forms of apples: different views, colors, shadings, shapes, and varieties.

Also keep in mind that the hierarchy shown above is a hierarchy of concepts. These recognizers are not physically placed above each other; because of the thin construction of the neocortex, it is physically only one pattern recognizer high. The conceptual hierarchy is created by the connections between the individual pattern recognizers.

An important attribute of the PRTM is how the recognitions are made inside each pattern recognition module. Stored in the module is a weight for each input dendrite indicating how important that input is to the recognition. The pattern recognizer has a threshold for firing (which indicates that this pattern recognizer has successfully recognized the pattern it is responsible for). Not every input pattern has to be present for a recognizer to fire. The recognizer may still fire if an input with a low weight is missing, but it is less likely to fire if a high-importance input is missing. When it fires, a pattern recognizer is basically saying, “The pattern I am responsible for is probably present.”

Successful recognition by a module of its pattern goes beyond just counting the input signals that are activated (even a count weighted by the importance parameter). The size (of each input) matters. There is another parameter (for each input) indicating the expected size of the input, and yet another indicating how variable that size is. To appreciate how this works, suppose we have a pattern recognizer that is responsible for recognizing the spoken word “steep.” This spoken word has four sounds: [s], [t], [E], and [p]. The [t] phoneme is what is known as a “dental consonant,” meaning that it is created by the tongue creating a burst of noise when air breaks its contact with the upper teeth. It is essentially impossible to articulate the [t] phoneme slowly. The [p] phoneme is considered a “plosive consonant” or “oral occlusive,” meaning that it is created when the vocal tract is suddenly blocked (by the lips in the case of [p]) so that air no longer passes. It is also necessarily quick. The [E] vowel is caused by resonances of the vocal cord and open mouth. It is considered a “long vowel,” meaning that it persists for a much longer period of time than consonants such as [t] and [p]; however, its duration can be quite variable. The [s] phoneme is known as a “sibilant consonant,” and is caused by the passage of air against the edges of the teeth, which are held close together. Its duration is typically shorter than that of a long vowel such as [E], but it is also variable (in other words, the [s] can be said quickly or you can drag it out).

In our work in speech recognition, we found that it is necessary to encode this type of information in order to recognize speech patterns. For example, the words “step” and “steep” are very similar. Although the [e] phoneme in “step” and the [E] in “steep” are somewhat different vowel sounds (in that they have different resonant frequencies), it is not reliable to distinguish these two words based on these often confusable vowel sounds. It is much more reliable to consider the observation that the [e] in “step” is relatively brief compared with the [E] in “steep.”

We can encode this type of information with two numbers for each input: the expected size and the degree of variability of that size. In our “steep” example, [t] and [p] would both have a very short expected duration as well as a small expected variability (that is, we do not expect to hear long t’s and p’s). The [s] sound would have a short expected duration but a larger variability because it is possible to drag it out. The [E] sound has a long expected duration as well as a high degree of variability.

In our speech examples, the “size” parameter refers to duration, but time is only one possible dimension. In our work in character recognition, we found that comparable spatial information was important in order to recognize printed letters (for example the dot over the letter “i” is expected to be much smaller than the portion under the dot). At much higher levels of abstraction, the neocortex will deal with patterns with all sorts of continuums, such as levels of attractiveness, irony, happiness, frustration, and myriad others. We can draw similarities across rather diverse continuums, as Darwin did when he related the physical size of geological canyons to the amount of differentiation among species.

In a biological brain, the source of these parameters comes from the brain’s own experience. We are not born with an innate knowledge of phonemes; indeed different languages have very different sets of them. This implies that multiple examples of a pattern are encoded in the learned parameters of each pattern recognizer (as it requires multiple instances of a pattern to ascertain the expected distribution of magnitudes of the inputs to the pattern). In some AI systems, these types of parameters are hand-coded by experts (for example, linguists who can tell us the expected durations of different phonemes, as I articulated above). In my own work, we found that having an AI system discover these parameters on its own from training data (similar to the way the brain does it) was a superior approach. Sometimes we used a hybrid approach; that is, we primed the system with the intuition of human experts (for the initial settings of the parameters) and then had the AI system automatically refine these estimates using a learning process from real examples of speech.

What the pattern recognition module is doing is computing the probability (that is, the likelihood based on all of its previous experience) that the pattern that it is responsible for recognizing is in fact currently represented by its active inputs. Each particular input to the module is active if the corresponding lower-level pattern recognizer is firing (meaning that that lower-level pattern was recognized). Each input also encodes the observed size (on some appropriate dimension such as temporal duration or physical magnitude or some other continuum) so that the size can be compared (with the stored size parameters for each input) by the module in computing the overall probability of the pattern.

How does the brain (and how can an AI system) compute the overall probability that the pattern (that the module is responsible for recognizing) is present given (1) the inputs (each with an observed size), (2) the stored parameters on size (the expected size and the variability of size) for each input, and (3) the parameters of the importance of each input? In the 1980s and 1990s, I and others pioneered a mathematical method called hierarchical hidden Markov models for learning these parameters and then using them to recognize hierarchical patterns. We used this technique in the recognition of human speech as well as the understanding of natural language. I describe this approach further in chapter 7.

Getting back to the flow of recognition from one level of pattern recognizers to the next, in the above example we see the information flow up the conceptual hierarchy from basic letter features to letters to words. Recognitions will continue to flow up from there to phrases and then more complex language structures. If we go up several dozen more levels, we get to higher-level concepts like irony and envy. Even though every pattern recognizer is working simultaneously, it does take time for recognitions to move upward in this conceptual hierarchy. Traversing each level takes between a few hundredths to a few tenths of a second to process. Experiments have shown that a moderately high-level pattern such as a face takes at least a tenth of a second. It can take as long as an entire second if there are significant distortions. If the brain were sequential (like conventional computers) and was performing each pattern recognition in sequence, it would have to consider every possible low-level pattern before moving on to the next level. Thus it would take many millions of cycles just to go through each level. That is exactly what happens when we simulate these processes on a computer. Keep in mind, however, that computers process millions of times faster than our biological circuits.

A very important point to note here is that information flows down the conceptual hierarchy as well as up. If anything, this downward flow is even more significant. If, for example, we are reading from left to right and have already seen and recognized the letters “A,” “P,” “P,” and “L,” the “APPLE” recognizer will predict that it is likely to see an “E” in the next position. It will send a signal down to the “E” recognizer saying, in effect, “Please be aware that there is a high likelihood that you will see your ‘E’ pattern very soon, so be on the lookout for it.” The “E” recognizer then adjusts its threshold such that it is more likely to recognize an “E.” So if an image appears next that is vaguely like an “E,” but is perhaps smudged such that it would not have been recognized as an “E” under “normal” circumstances, the “E” recognizer may nonetheless indicate that it has indeed seen an “E,” since it was expected.

The neocortex is, therefore, predicting what it expects to encounter. Envisaging the future is one of the primary reasons we have a neocortex. At the highest conceptual level, we are continually making predictions—who is going to walk through the door next, what someone is likely to say next, what we expect to see when we turn the corner, the likely results of our own actions, and so on. These predictions are constantly occurring at every level of the neocortex hierarchy. We often misrecognize people and things and words because our threshold for confirming an expected pattern is too low.

In addition to positive signals, there are also negative or inhibitory signals which indicate that a certain pattern is less likely to exist. These can come from lower conceptual levels (for example, the recognition of a mustache will inhibit the likelihood that a person I see in the checkout line is my wife), or from a higher level (for example, I know that my wife is on a trip, so the person in the checkout line can’t be she). When a pattern recognizer receives an inhibitory signal, it raises the recognition threshold, but it is still possible for the pattern to fire (so if the person in line really is her, I may still recognize her).

The Nature of the Data Flowing into a Neocortical Pattern Recognizer

Let’s consider further what the data for a pattern looks like. If the pattern is a face, the data exists in at least two dimensions. We cannot say that the eyes necessarily come first, followed by the nose, and so on. The same thing is true for most sounds. A musical piece has at least two dimensions. There may be more than one instrument and/or voice making sounds at the same time. Moreover, a single note of a complex instrument such as the piano consists of multiple frequencies. A single human voice consists of varying levels of energy in dozens of different frequency bands simultaneously. So a pattern of sound may be complex at any one instant, and these complex instants stretch out over time. Tactile inputs are also two-dimensional, since the skin is a two-dimensional sense organ, and such patterns may change over the third dimension of time.

So it would seem that the input to a neocortex pattern processor must comprise two- if not three-dimensional patterns. However, we can see in the structure of the neocortex that the pattern inputs are only one-dimensional lists. All of our work in the field of creating artificial pattern recognition systems (such as speech recognition and visual recognition systems) demonstrates that we can (and did) represent two- and three-dimensional phenomena with such one-dimensional lists. I’ll describe how these methods work in chapter 7, but for now we can proceed with the understanding that the input to each pattern processor is a one-dimensional list, even though the pattern itself may inherently reflect more than one dimension.

We should factor in at this point the insight that the patterns we have learned to recognize (for example, a specific dog or the general idea of a “dog,” a musical note or a piece of music) are exactly the same mechanism that is the basis for our memories. Our memories are in fact patterns organized as lists (where each item in each list is another pattern in the cortical hierarchy) that we have learned and then recognize when presented with the appropriate stimulus. In fact, memories exist in the neocortex in order to be recognized.

The only exception to this is at the lowest possible conceptual level, in which the input data to a pattern represents specific sensory information (for example, image data from the optic nerve). Even this lowest level of pattern, however, has been significantly transformed into simple patterns by the time it reaches the cortex. The lists of patterns that constitute a memory are in forward order, and we are able to remember our memories only in that order, hence the difficulty we have in reversing our memories.

A memory needs to be triggered by another thought/memory (these are the same thing). We can experience this mechanism of triggering when we are perceiving a pattern. When we perceived “A,” “P,” “P,” and “L,” the “A P P L E” pattern predicted that we would see an “E” and triggered the “E” pattern that it is now expected. Our cortex is thereby “thinking” of seeing an “E” even before we see it. If this particular interaction in our cortex has our attention, we will think about “E” before we see it or even if we never see it. A similar mechanism triggers old memories. Usually there is an entire chain of such links. Even if we do have some level of awareness of the memories (that is, the patterns) that triggered the old memory, memories (patterns) do not have language or image labels. This is the reason why old memories may seem to suddenly jump into our awareness. Having been buried and not activated for perhaps years, they need a trigger in the same way that a Web page needs a Web link to be activated. And just as a Web page can become “orphaned” because no other page links to it, the same thing can happen to our memories.

Our thoughts are largely activated in one of two modes, undirected and directed, both of which use these same cortical links. In the undirected mode, we let the links play themselves out without attempting to move them in any particular direction. Some forms of meditation (such as Transcendental Meditation, which I practice) are based on letting the mind do exactly this. Dreams have this quality as well.

In directed thinking we attempt to step through a more orderly process of recalling a memory (a story, for example) or solving a problem. This also involves stepping through lists in our neocortex, but the less structured flurry of undirected thought will also accompany the process. The full content of our thinking is therefore very disorderly, a phenomenon that James Joyce illuminated in his “stream of consciousness” novels.

As you think through the memories/stories/patterns in your life, whether they involve a chance encounter with a mother with a baby carriage and baby on a walk or the more important narrative of how you met your spouse, your memories consist of a sequence of patterns. Because these patterns are not labeled with words or sounds or pictures or videos, when you try to recall a significant event, you will essentially be reconstructing the images in your mind, because the actual images do not exist.

If we were to “read” the mind of someone and peer at exactly what is going on in her neocortex, it would be very difficult to interpret her memories, whether we were to take a look at patterns that are simply stored in the neocortex waiting to be triggered or those that have been triggered and are currently being experienced as active thoughts. What we would “see” is the simultaneous activation of millions of pattern recognizers. A hundredth of a second later, we would see a different set of a comparable number of activated pattern recognizers. Each such pattern would be a list of other patterns, and each of those patterns would be a list of other patterns, and so on until we reached the most elementary simple patterns at the lowest level. It would be extremely difficult to interpret what these higher-level patterns meant without actually copying all of the information at every level into our own cortex. Thus each pattern in our neocortex is meaningful only in light of all the information carried in the levels below it. Moreover, other patterns at the same level and at higher levels are also relevant in interpreting a particular pattern because they provide context. True mind reading, therefore, would necessitate not just detecting the activations of the relevant axons in a person’s brain, but examining essentially her entire neocortex with all of its memories to understand these activations.

As we experience our own thoughts and memories, we “know” what they mean, but they do not exist as readily explainable thoughts and recollections. If we want to share them with others, we need to translate them into language. This task is also accomplished by the neocortex, using pattern recognizers trained with patterns that we have learned for the purpose of using language. Language is itself highly hierarchical and evolved to take advantage of the hierarchical nature of the neocortex, which in turn reflects the hierarchical nature of reality. The innate ability of humans to learn the hierarchical structures in language that Noam Chomsky wrote about reflects the structure of the neocortex. In a 2002 paper he coauthored, Chomsky cites the attribute of “recursion” as accounting for the unique language faculty of the human species.⁴ Recursion, according to Chomsky, is the ability to put together small parts into a larger chunk, and then use that chunk as a part in yet another structure, and to continue this process iteratively. In this way we are able to build the elaborate structures of sentences and paragraphs from a limited set of words. Although Chomsky was not explicitly referring here to brain structure, the capability he is describing is exactly what the neocortex does.

Lower species of mammals largely use up their neocortex with the challenges of their particular lifestyles. The human species acquired additional capacities by having grown substantially more cortex to handle spoken and written language. Some people have learned such skills better than others. If we have told a particular story many times, we will begin to actually learn the sequence of language that describes the story as a series of separate sequences. Even in this case our memory is not a strict sequence of words, but rather of language structures that we need to translate into specific word sequences each time we deliver the story. That is why we tell a story a bit differently each time we share it (unless we learn the exact word sequence as a pattern).

For each of these descriptions of specific thought processes, we also need to consider the issue of redundancy. As I mentioned, we don’t have a single pattern representing the important entities in our lives, whether those entities constitute sensory categories, language concepts, or memories of events. Every important pattern—at every level—is repeated many times. Some of these recurrences represent simple repetitions, whereas many represent different perspectives and vantage points. This is a principal reason why we can recognize a familiar face from various orientations and under a range of lighting conditions. Each level up the hierarchy has substantial redundancy, allowing sufficient variability that is consistent with that concept.

So if we were to imagine examining your neocortex when you were looking at a particular loved one, we would see a great many firings of the axons of the pattern recognizers at every level, from the basic level of primitive sensory patterns up to many different patterns representing that loved one’s image. We would also see massive numbers of firings representing other aspects of the situation, such as that person’s movements, what she is saying, and so on. So if the experience seems much richer than just an orderly trip up a hierarchy of features, it is.

A computer simulation of the firings of many simultaneous pattern recognizers in the neocortex.

But the basic mechanism of going up a hierarchy of pattern recognizers in which each higher conceptual level represents a more abstract and more integrated concept remains valid. The flow of information downward is even greater, as each activated level of recognized pattern sends predictions to the next lower-level pattern recognizer of what it is likely to be encountering next. The apparent lushness of human experience is a result of the fact that all of the hundreds of millions of pattern recognizers in our neocortex are considering their inputs simultaneously.

In chapter 5 I’ll discuss the flow of information from touch, vision, hearing, and other sensory organs into the neocortex. These early inputs are processed by cortical regions that are devoted to relevant types of sensory input (although there is enormous plasticity in the assignment of these regions, reflecting the basic uniformity of function in the neocortex). The conceptual hierarchy continues above the highest concepts in each sensory region of the neocortex. The cortical association areas integrate input from the different sensory inputs. When we hear something that perhaps sounds like our spouse’s voice, and then see something that is perhaps indicative of her presence, we don’t engage in an elaborate process of logical deduction; rather, we instantly perceive that our spouse is present from the combination of these sensory recognitions. We integrate all of the germane sensory and perceptual cues—perhaps even the smell of her perfume or his cologne—as one multilevel perception.

At a conceptual level above the cortical sensory association areas, we are capable of dealing with—perceiving, remembering, and thinking about—even more abstract concepts. At the highest level we recognize patterns such as that’s funny, or she’s pretty, or that’s ironic, and so on. Our memories include these abstract recognition patterns as well. For example, we might recall that we were taking a walk with someone and that she said something funny, and we laughed, though we may not remember the actual joke itself. The memory sequence for that recollection has simply recorded the perception of humor but not the precise content of what was funny.

In the previous chapter I noted that we can often recognize a pattern even though we don’t recognize it well enough to be able to describe it. For example, I believe I could pick out a picture of the woman with the baby carriage whom I saw earlier today from among a group of pictures of other women, despite the fact that I am unable to actually visualize her and cannot describe much specific about her. In this case my memory of her is a list of certain high-level features. These features do not have language or image labels attached to them, and they are not pixel images, so while I am able to think about her, I am unable to describe her. However, if I am presented with a picture of her, I can process the image, which results in the recognition of the same high-level features that were recognized the first time I saw her. I would be able to thereby determine that the features match and thus confidently pick out her picture.

Even though I saw this woman only once on my walk, there are probably already multiple copies of her pattern in my neocortex. However, if I don’t think about her for a given period of time, then these pattern recognizers will become reassigned to other patterns. That is why memories grow dimmer with time: The amount of redundancy becomes reduced until certain memories become extinct. However, now that I have memorialized this particular woman by writing about her here, I probably won’t forget her so easily.

Autoassociation and Invariance

In the previous chapter I discussed how we can recognize a pattern even if the entire pattern is not present, and also if it is distorted. The first capability is called autoassociation: the ability to associate a pattern with a part of itself. The structure of each pattern recognizer inherently supports this capability.

As each input from a lower-level pattern recognizer flows up to a higher-level one, the connection can have a “weight,” indicating how important that particular element in the pattern is. Thus the more significant elements of a pattern are more heavily weighted in considering whether that pattern should trigger as “recognized.” Lincoln’s beard, Elvis’s sideburns, and Einstein’s famous tongue gesture are likely to have high weights in the patterns we’ve learned about the appearance of these iconic figures. The pattern recognizer computes a probability that takes the importance parameters into account. Thus the overall probability is lower if one or more of the elements is missing, though the threshold of recognition may nonetheless be met. As I pointed out, the computation of the overall probability (that the pattern is present) is more complicated than a simple weighted sum in that the size parameters also need to be considered.

If the pattern recognizer has received a signal from a higher-level recognizer that its pattern is “expected,” then the threshold is effectively lowered (that is, made easier to achieve). Alternatively, such a signal may simply add to the total of the weighted inputs, thereby compensating for a missing element. This happens at every level, so that a pattern such as a face that is several levels up from the bottom may be recognized even with multiple missing features.

The ability to recognize patterns even when aspects of them are transformed is called feature invariance, and is dealt with in four ways. First, there are global transformations that are accomplished before the neocortex receives sensory data. We will discuss the voyage of sensory data from the eyes, ears, and skin in the section “The Sensory Pathway” on page 94.

The second method takes advantage of the redundancy in our cortical pattern memory. Especially for important items, we have learned many different perspectives and vantage points for each pattern. Thus many variations are separately stored and processed.

The third and most powerful method is the ability to combine two lists. One list can have a set of transformations that we have learned may apply to a certain category of pattern; the cortex will apply this same list of possible changes to another pattern. That is how we understand such language phenomena as metaphors and similes.

For example, we have learned that certain phonemes (the basic sounds of language) may be missing in spoken speech (for example, “goin’”). If we then learn a new spoken word (for example, “driving”), we will be able to recognize that word if one of its phonemes is missing even if we have never experienced that word in that form before, because we have become familiar with the general phenomenon of certain phonemes being omitted. As another example, we may learn that a particular artist likes to emphasize (by making larger) certain elements of a face, such as the nose. We can then identify a face with which we are familiar to which that modification has been applied even if we have never seen that modification on that face. Certain artistic modifications emphasize the very features that are recognized by our pattern recognition–based neocortex. As mentioned, that is precisely the basis of caricature.

The fourth method derives from the size parameters that allow a single module to encode multiple instances of a pattern. For example, we have heard the word “steep” many times. A particular pattern recognition module that is recognizing this spoken word can encode these multiple examples by indicating that the duration of [E] has a high expected variability. If all the modules for words including [E] share a similar phenomenon, that variability could be encoded in the models for [E] itself. However, different words incorporating [E] (or many other phonemes) may have different amounts of expected variability. For example, the word “peak” is likely not to have the [E] phoneme as drawn out as in the word “steep.”

Learning

So far we have examined how we recognize (sensory and perceptual) patterns and recall sequences of patterns (our memory of things, people, and events). However, we are not born with a neocortex filled with any of these patterns. Our neocortex is virgin territory when our brain is created. It has the capability of learning and therefore of creating connections between its pattern recognizers, but it gains those connections from experience.

This learning process begins even before we are born, occurring simultaneously with the biological process of actually growing a brain. A fetus already has a brain at one month, although it is essentially a reptile brain, as the fetus actually goes through a high-speed re-creation of biological evolution in the womb. The natal brain is distinctly a human brain with a human neocortex by the time it reaches the third trimester of pregnancy. At this time the fetus is having experiences, and the neocortex is learning. She can hear sounds, especially her mother’s heartbeat, which is one likely reason that the rhythmic qualities of music are universal to human culture. Every human civilization ever discovered has had music as part of its culture, which is not the case with other art forms, such as pictorial art. It is also the case that the beat of music is comparable to our heart rate. Music beats certainly vary—otherwise music would not keep our interest—but heartbeats vary also. An overly regular heartbeat is actually a symptom of a diseased heart. The eyes of a fetus are partially open twenty-six weeks after conception, and are fully open most of the time by twenty-eight weeks after conception. There may not be much to see inside the womb, but there are patterns of light and dark that the neocortex begins to process.

So while a newborn baby has had a bit of experience in the womb, it is clearly limited. The neocortex may also learn from the old brain (a topic I discuss in chapter 5), but in general at birth the child has a lot to learn—everything from basic primitive sounds and shapes to metaphors and sarcasm.

Learning is critical to human intelligence. If we were to perfectly model and simulate the human neocortex (as the Blue Brain Project is attempting to do) and all of the other brain regions that it requires to function (such as the hippocampus and thalamus), it would not be able to do very much—in the same way that a newborn infant cannot do much (other than to be cute, which is definitely a key survival adaptation).

Learning and recognition take place simultaneously. We start learning immediately, and as soon as we’ve learned a pattern, we immediately start recognizing it. The neocortex is continually trying to make sense of the input presented to it. If a particular level is unable to fully process and recognize a pattern, it gets sent to the next higher level. If none of the levels succeeds in recognizing a pattern, it is deemed to be a new pattern. Classifying a pattern as new does not necessarily mean that every aspect of it is new. If we are looking at the paintings of a particular artist and see a cat’s face with the nose of an elephant, we will be able to identify each of the distinctive features but will notice that this combined pattern is something novel, and are likely to remember it. Higher conceptual levels of the neocortex, which understand context—for example, the circumstance that this picture is an example of a particular artist’s work and that we are attending an opening of a showing of new paintings by that artist—will note the unusual combination of patterns in the cat-elephant face but will also include these contextual details as additional memory patterns.

New memories such as the cat-elephant face are stored in an available pattern recognizer. The hippocampus plays a role in this process, and we’ll discuss what is known about the actual biological mechanisms in the following chapter. For the purposes of our neocortex model, it is sufficient to say that patterns that are not otherwise recognized are stored as new patterns and are appropriately connected to the lower-level patterns that form them. The cat-elephant face, for example, will be stored in several different ways: The novel arrangement of facial parts will be stored as well as contextual memories that include the artist, the situation, and perhaps the fact that we laughed when we first saw it.

Memories that are successfully recognized may also result in the creation of a new pattern to achieve greater redundancy. If patterns are not perfectly recognized, they are likely to be stored as reflecting a different perspective of the item that was recognized.

What, then, is the overall method for determining what patterns get stored? In mathematical terms, the problem can be stated as follows: Using the available limits of pattern storage, how do we optimally represent the input patterns that have thus far been presented? While it makes sense to allow for a certain amount of redundancy, it would not be practical to fill up the entire available storage area (that is, the entire neocortex) with repeated patterns, as that would not allow for a sufficient diversity of patterns. A pattern such as the [E] phoneme in spoken words is something we have experienced countless times. It is a simple pattern of sound frequencies and it undoubtedly enjoys significant redundancy in our neocortex. We could fill up our entire neocortex with repeated patterns of the [E] phoneme. There is a limit, however, to useful redundancy, and a common pattern such as this clearly has reached it.

There is a mathematical solution to this optimization problem called linear programming, which solves for the best possible allocation of limited resources (in this case, a limited number of pattern recognizers) that would represent all of the cases on which the system has trained. Linear programming is designed for systems with one-dimensional inputs, which is another reason why it is optimal to represent the input to each pattern recognition module as a linear string of inputs. We can use this mathematical approach in a software system, and though an actual brain is further constrained by the physical connections it has available that it can adapt between pattern recognizers, the method is nonetheless similar.

An important implication of this optimal solution is that experiences that are routine are recognized but do not result in a permanent memory’s being made. With regard to my walk, I experienced millions of patterns at every level, from basic visual edges and shadings to objects such as lampposts and mailboxes and people and animals and plants that I passed. Almost none of what I experienced was unique, and the patterns that I recognized had long since reached their optimal level of redundancy. The result is that I recall almost nothing from this walk. The few details that I do remember are likely to get overwritten with new patterns by the time I take another few dozen walks—except for the fact that I have now memorialized this particular walk by writing about it.

One important point that applies to both our biological neocortex and attempts to emulate it is that it is difficult to learn too many conceptual levels simultaneously. We can essentially learn one or at most two conceptual levels at a time. Once that learning is relatively stable, we can go on to learn the next level. We may continue to fine-tune the learning in the lower levels, but our learning focus is on the next level of abstraction. This is true at both the beginning of life, as newborns struggle with basic shapes, and later in life, as we struggle to learn new subject matter, one level of complexity at a time. We find the same phenomenon in machine emulations of the neocortex. However, if they are presented increasingly abstract material one level at a time, machines are capable of learning just as humans do (although not yet with as many conceptual levels).

The output of a pattern can feed back to a pattern at a lower level or even to the pattern itself, giving the human brain its powerful recursive ability. An element of a pattern can be a decision point based on another pattern. This is especially useful for lists that compose actions—for example, getting another tube of toothpaste if the current one is empty. These conditionals exist at every level. As anyone who has attempted to program a procedure on a computer knows, conditionals are vital to describing a course of action.

The Language of Thought

To summarize what we’ve learned so far about the way the neocortex works, please refer to the diagram of the neocortical pattern recognition module on page 42.

a) Dendrites enter the module that represents the pattern. Even though patterns may seem to have two- or three-dimensional qualities, they are represented by a one-dimensional sequence of signals. The pattern must be present in this (sequential) order for the pattern recognizer to be able to recognize it. Each of the dendrites is connected ultimately to one or more axons of pattern recognizers at a lower conceptual level that have recognized a lower-level pattern that constitutes part of this pattern. For each of these input patterns, there may be many lower-level pattern recognizers that can generate the signal that the lower-level pattern has been recognized. The necessary threshold to recognize the pattern may be achieved even if not all of the inputs have signaled. The module computes the probability that the pattern it is responsible for is present. This computation considers the “importance” and “size” parameters (see [f] below).

Note that some of the dendrites transmit signals into the module and some out of the module. If all of the input dendrites to this pattern recognizer are signaling that their lower-level patterns have been recognized except for one or two, then this pattern recognizer will send a signal down to the pattern recognizer(s) recognizing the lower-level patterns that have not yet been recognized, indicating that there is a high likelihood that that pattern will soon be recognized and that lower-level recognizer(s) should be on the lookout for it.

b) When this pattern recognizer recognizes its pattern (based on all or most of the input dendrite signals being activated), the axon (output) of this pattern recognizer will activate. In turn, this axon can connect to an entire network of dendrites connecting to many higher-level pattern recognizers that this pattern is input to. This signal will transmit magnitude information so that the pattern recognizers at the next higher conceptual level can consider it.

c) If a higher-level pattern recognizer is receiving a positive signal from all or most of its constituent patterns except for the one represented by this pattern recognizer, then that higher-level recognizer might send a signal down to this recognizer indicating that its pattern is expected. Such a signal would cause this pattern recognizer to lower its threshold, meaning that it would be more likely to send a signal on its axon (indicating that its pattern is considered to have been recognized) even if some of its inputs are missing or unclear.

d) Inhibitory signals from below would make it less likely that this pattern recognizer will recognize its pattern. This can result from recognition of lower-level patterns that are inconsistent with the pattern associated with this pattern recognizer (for example, recognition of a mustache by a lower-level recognizer would make it less likely that this image is “my wife”).

e) Inhibitory signals from above would also make it less likely that this pattern recognizer will recognize its pattern. This can result from a higher-level context that is inconsistent with the pattern associated with this recognizer.

f) For each input, there are stored parameters for importance, expected size, and expected variability of size. The module computes an overall probability that the pattern is present based on all of these parameters and the current signals indicating which of the inputs are present and their magnitudes. A mathematically optimal way to accomplish this is with a technique called hidden Markov models. When such models are organized in a hierarchy (as they are in the neocortex or in attempts to simulate a neocortex), we call them hierarchical hidden Markov models.

Patterns triggered in the neocortex trigger other patterns. Partially complete patterns send signals down the conceptual hierarchy; completed patterns send signals up the conceptual hierarchy. These neocortical patterns are the language of thought. Just like language, they are hierarchical, but they are not language per se. Our thoughts are not conceived primarily in the elements of language, although since language also exists as hierarchies of patterns in our neocortex, we can have language-based thoughts. But for the most part, thoughts are represented in these neocortical patterns.

As I discussed above, if we were able to detect the pattern activations in someone’s neocortex, we would still have little idea what those pattern activations meant without also having access to the entire hierarchy of patterns above and below each activated pattern. That would pretty much require access to that person’s entire neocortex. It is hard enough for us to understand the content of our own thoughts, but understanding another person’s requires mastering a neocortex different from our own. Of course we don’t yet have access to someone else’s neocortex; we need instead to rely on her attempts to express her thoughts into language (as well as other means such as gestures). People’s incomplete ability to accomplish these communication tasks adds another layer of complexity—it is no wonder that we misunderstand one another as much as we do.

We have two modes of thinking. One is nondirected thinking, in which thoughts trigger one another in a nonlogical way. When we experience a sudden recollection of a memory from years or decades ago while doing something else, such as raking the leaves or walking down the street, the experience is recalled—as all memories are—as a sequence of patterns. We do not immediately visualize the scene unless we can call upon a lot of other memories that enable us to synthesize a more robust recollection. If we do visualize the scene in that way, we are essentially creating it in our mind from hints at the time of recollection; the memory itself is not stored in the form of images or visualizations. As I mentioned earlier, the triggers that led this thought to pop into our mind may or may not be evident. The sequence of relevant thoughts may have been immediately forgotten. Even if we do remember it, it will be a nonlinear and circuitous sequence of associations.

The second mode of thinking is directed thinking, which we use when we attempt to solve a problem or formulate an organized response. For example, we might be rehearsing in our mind something we plan to say to someone, or we might be formulating a passage we want to write (in a book on the mind, perhaps). As we think about tasks such as these, we have already broken down each one into a hierarchy of subtasks. Writing a book, for example, involves writing chapters; each chapter has sections; each section has paragraphs; each paragraph contains sentences that express ideas; each idea has its configuration of elements; each element and each relationship between elements is an idea that needs to be articulated; and so on. At the same time, our neocortical structures have learned certain rules that should be followed. If the task is writing, then we should try to avoid unnecessary repetition; we should try to make sure that the reader can follow what is being written; we should try to follow rules about grammar and style; and so on. The writer needs therefore to build a model of the reader in his mind, and that construct is hierarchical as well. In doing directed thinking, we are stepping through lists in our neocortex, each of which expands into extensive hierarchies of sublists, each with its own considerations. Keep in mind that elements in a list in a neocortical pattern can include conditionals, so our subsequent thoughts and actions will depend on assessments made as we go through the process.

Moreover, each such directed thought will trigger hierarchies of undirected thoughts. A continual storm of ruminations attends both our sensory experiences and our attempts at directed thinking. Our actual mental experience is complex and messy, made up of these lightning storms of triggered patterns, which change about a hundred times a second.

The Language of Dreams

Dreams are examples of undirected thoughts. They make a certain amount of sense because the phenomenon of one thought’s triggering another is based on the actual linkages of patterns in our neocortex. To the extent that a dream does not make sense, we attempt to fix it through our ability to confabulate. As I will describe in chapter 9, split-brain patients (whose corpus callosum, which connects the two hemispheres of the brain, is severed or damaged) will confabulate (make up) explanations with their left brain—which controls the speech center—to explain what the right brain just did with input that the left brain did not have access to. We confabulate all the time in explaining the outcome of events. If you want a good example of this, just tune in to the daily commentary on the movement of financial markets. No matter how the markets perform, it’s always possible to come up with a good explanation for why it happened, and such after-the-fact commentary is plentiful. Of course, if these commentators really understood the markets, they wouldn’t have to waste their time doing commentary.

The act of confabulating is of course also done in the neocortex, which is good at coming up with stories and explanations that meet certain constraints. We do that whenever we retell a story. We will fill in details that may not be available or that we may have forgotten so that the story makes more sense. That is why stories change over time as they are told over and over again by new storytellers with perhaps different agendas. As spoken language led to written language, however, we had a technology that could record a definitive version of a story and prevent this sort of drift.

The actual content of a dream, to the extent that we remember it, is again a sequence of patterns. These patterns represent constraints in a story; we then confabulate a story that fits these constraints. The version of the dream that we retell (even if only to ourselves silently) is this confabulation. As we recount a dream we trigger cascades of patterns that fill in the actual dream as we originally experienced it.

There is one key difference between dream thoughts and our thinking while awake. One of the lessons we learn in life is that certain actions, even thoughts, are not permissible in the real world. For example, we learn that we cannot immediately fulfill our desires. There are rules against grabbing the money in the cash register at a store, and constraints on interacting with a person to whom we may be physically attracted. We also learn that certain thoughts are not permissible because they are culturally forbidden. As we learn professional skills, we learn the ways of thinking that are recognized and rewarded in our professions, and thereby avoid patterns of thought that might betray the methods and norms of that profession. Many of these taboos are worthwhile, as they enforce social order and consolidate progress. However, they can also prevent progress by enforcing an unproductive orthodoxy. Such orthodoxy is precisely what Einstein left behind when he tried to ride a light beam with his thought experiments.

Cultural rules are enforced in the neocortex with help from the old brain, especially the amygdala. Every thought we have triggers other thoughts, and some of them will relate to associated dangers. We learn, for example, that breaking a cultural norm even in our private thoughts can lead to ostracism, which the neocortex realizes threatens our well-being. If we entertain such thoughts, the amygdala is triggered, and that generates fear, which generally leads to terminating that thought.

In dreams, however, these taboos are relaxed, and we will often dream about matters that are culturally, sexually, or professionally forbidden. It is as if our brain realizes that we are not an actual actor in the world while dreaming. Freud wrote about this phenomenon but also noted that we will disguise such dangerous thoughts, at least when we attempt to recall them, so that the awake brain continues to be protected from them.

Relaxing professional taboos turns out to be useful for creative problem solving. I use a mental technique each night in which I think about a particular problem before I go to sleep. This triggers sequences of thoughts that will continue into my dreams. Once I am dreaming, I can think—dream—about solutions to the problem without the burden of the professional restraints I carry during the day. I can then access these dream thoughts in the morning while in an in-between state of dreaming and being awake, sometimes referred to as “lucid dreaming.”⁵

Freud also famously wrote about the ability to gain insight into a person’s psychology by interpreting dreams. There is of course a vast literature on all aspects of this theory, but the fundamental notion of gaining insight into ourselves through examination of our dreams makes sense. Our dreams are created by our neocortex, and thus their substance can be revealing of the content and connections found there. The relaxation of the constraints on our thinking that exist while we are awake is also useful in revealing neocortical content that we otherwise would be unable to access directly. It is also reasonable to conclude that the patterns that end up in our dreams represent important matters to us and thereby clues in understanding our unresolved desires and fears.

The Roots of the Model

As I mentioned above, I led a team in the 1980s and 1990s that developed the technique of hierarchical hidden Markov models to recognize human speech and understand natural-language statements. This work was the predecessor to today’s widespread commercial systems that recognize and understand what we are trying to tell them (car navigation systems that you can talk to, Siri on the iPhone, Google Voice Search, and many others). The technique we developed had substantially all of the attributes that I describe in the PRTM. It included a hierarchy of patterns with each higher level being conceptually more abstract than the one below it. For example, in speech recognition the levels included basic patterns of sound frequency at the lowest level, then phonemes, then words and phrases (which were often recognized as if they were words). Some of our speech recognition systems could understand the meaning of natural-language commands, so yet higher levels included such structures as noun and verb phrases. Each pattern recognition module could recognize a linear sequence of patterns from a lower conceptual level. Each input had parameters for importance, size, and variability of size. There were “downward” signals indicating that a lower-level pattern was expected. I discuss this research in more detail in chapter 7.

In 2003 and 2004, PalmPilot inventor Jeff Hawkins and Dileep George developed a hierarchical cortical model called hierarchical temporal memory. With science writer Sandra Blakeslee, Hawkins described this model eloquently in their book On Intelligence. Hawkins provides a strong case for the uniformity of the cortical algorithm and its hierarchical and list-based organization. There are some important differences between the model presented in On Intelligence and what I present in this book. As the name implies, Hawkins is emphasizing the temporal (time-based) nature of the constituent lists. In other words, the direction of the lists is always forward in time. His explanation for how the features in a two-dimensional pattern such as the printed letter “A” have a direction in time is predicated on eye movement. He explains that we visualize images using saccades, which are very rapid movements of the eye of which we are unaware. The information reaching the neocortex is therefore not a two-dimensional set of features but rather a time-ordered list. While it is true that our eyes do make very rapid movements, the sequence in which they view the features of a pattern such as the letter “A” does not always occur in a consistent temporal order. (For example, eye saccades will not always register the top vertex in “A” before its bottom concavity.) Moreover, we can recognize a visual pattern that is presented for only a few tens of milliseconds, which is too short a period of time for eye saccades to scan it. It is true that the pattern recognizers in the neocortex store a pattern as a list and that the list is indeed ordered, but the order does not necessarily represent time. That is often indeed the case, but it may also represent a spatial or higher-level conceptual ordering as I discussed above.

The most important difference is the set of parameters that I have included for each input into the pattern recognition module, especially the size and size variability parameters. In the 1980s we actually tried to recognize human speech without this type of information. This was motivated by linguists’ telling us that the duration information was not especially important. This perspective is illustrated by dictionaries that write out the pronunciation of each word as a string of phonemes, for example the word “steep” as [s] [t] [E] [p], with no indication of how long each phoneme is expected to last. The implication is that if we create programs to recognize phonemes and then encounter this particular sequence of four phonemes (in a spoken utterance), we should be able to recognize that spoken word. The system we built using this approach worked to some extent but not well enough to deal with such attributes as a large vocabulary, multiple speakers, and words spoken continuously without pauses. When we used the technique of hierarchical hidden Markov models in order to incorporate the distribution of magnitudes of each input, performance soared.