The Filter Bubble

You have one identity,” Facebook founder Mark Zuckerberg told journalist David Kirkpatrick for his book The Facebook Effect. “The days of you having a different image for your work friends or coworkers and for the other people you know are probably coming to an end pretty quickly…. Having two identities for yourself is an example of a lack of integrity.”

A year later, soon after the book had been published, twenty-six-year-old Zuckerberg sat onstage with Kirkpatrick and NPR interviewer Guy Raz at the Computer History Museum in Mountain View, California. “In David’s book,” Raz said, “you say that people should have one identity…. But I behave a different way around my family than I do around my colleagues.”

Zuckerberg shrugged. “No, I think that was just a sentence I said.”

Raz continued: “Are you the same person right now as when you’re with your friends?”

“Uh, yeah,” Zuckerberg said. “Same awkward self.”

If Mark Zuckerberg were a standard mid-twenty-something, this tangle of views might be par for the course: Most of us don’t spend too much time musing philosophically about the nature of identity. But Zuckerberg controls the world’s most powerful and widely used technology for managing and expressing who we are. And his views on the matter are central to his vision for the company and for the Internet.

Speaking at an event during New York’s Ad Week, Facebook COO Sheryl Sandberg said she expected the Internet to change quickly. “People don’t want something targeted to the whole world—they want something that reflects what they want to see and know,” she said, suggesting that in three to five years that would be the norm. Facebook’s goal is to be at the center of that process—the singular platform through which every other service and Web site incorporates your personal and social data. You have one identity, it’s your Facebook identity, and it colors your experience everywhere you go.

It’s hard to imagine a more dramatic departure from the early days of the Internet, in which not exposing your identity was part of the appeal. In chat rooms and online forums, your gender, race, age, and location were whatever you said they were, and the denizens of these spaces exulted about the way the medium allowed you to shed your skin. Electronic Frontier Foundation (EFF) founder John Perry Barlow dreamed of “creating a world that all may enter without privilege or prejudice accorded by race, economic power, military force, or station of birth.” The freedom that this offered anyone who was interested to transgress and explore, to try on different personas for size, felt revolutionary.

As law and commerce have caught up with technology, however, the space for anonymity online is shrinking. You can’t hold an anonymous person responsible for his or her actions: Anonymous customers commit fraud, anonymous commenters start flame wars, and anonymous hackers cause trouble. To establish the trust that community and capitalism are built on, you need to know whom you’re dealing with.

As a result, there are dozens of companies working on deanonymizing the Web. PeekYou, a firm founded by the creator of RateMyProfessors.com, is patenting ways of connecting online activities done under a pseudonym with the real name of the person involved. Another company, Phorm, helps Internet service providers use a method called “deep packet inspection” to analyze the traffic that flows through their servers; Phorm aims to build nearly comprehensive profiles of each customer to use for advertising and personalized services. And if ISPs are leery, BlueCava is compiling a database of every computer, smartphone, and online-enabled gadget in the world, which can be tied to the individual people who use them. Even if you’re using the highest privacy settings in your Web browser, in other words, your hardware may soon give you away.

These technological developments pave the way for a more persistent kind of personalization than anything we’ve experienced to date. It also means that we’ll increasingly be forced to trust the companies at the center of this process to properly express and synthesize who we really are. When you meet someone in a bar or a park, you look at how they behave and act and form an impression accordingly. Facebook and the other identity services aim to mediate that process online; if they don’t do it right, things can get fuzzy and distorted. To personalize well, you have to have the right idea of what represents a person.

There’s another tension in the interplay of identity and personalization. Most personalized filters are based on a three-step model. First, you figure out who people are and what they like. Then, you provide them with content and services that best fit them. Finally, you tune to get the fit just right. Your identity shapes your media. There’s just one flaw in this logic: Media also shape identity. And as a result, these services may end up creating a good fit between you and your media by changing… you. If a self-fulfilling prophecy is a false definition of the world that through one’s actions becomes true, we’re now on the verge of self-fulfilling identities, in which the Internet’s distorted picture of us becomes who we really are.

Personalized filtering can even affect your ability to choose your own destiny. In “Of Sirens and Amish Children,” a muchcited tract, information law theorist Yochai Benkler describes how more-diverse information sources make us freer. Autonomy, Benkler points out, is a tricky concept: To be free, you have to be able not only to do what you want, but to know what’s possible to do. The Amish children in the title are plaintiffs in a famous court case, Wisconsin v. Yoder, whose parents sought to prevent them from attending public school so that they wouldn’t be exposed to modern life. Benkler argues that this is a real threat to the children’s freedom: Not knowing that it’s possible to be an astronaut is just as much a prohibition against becoming one as knowing and being barred from doing so.

Of course, too many options are just as problematic as too few—you can find yourself overwhelmed by the number of options or paralyzed by the paradox of choice. But the basic point remains: The filter bubble doesn’t just reflect your identity. It also illustrates what choices you have. Students who go to Ivy League colleges see targeted advertisements for jobs that students at state schools are never even aware of. The personal feeds of professional scientists might feature articles about contests that amateurs never become aware of. By illustrating some possibilities and blocking out others, the filter bubble has a hand in your decisions. And in turn, it shapes who you become.

A Bad Theory of You

The way that personalization shapes identity is still becoming clear—especially because most of us still spend more time consuming broadcast media than personalized content streams. But by looking at how the major filterers think about identity, it’s becoming possible to predict what these changes might look like. Personalization requires a theory of what makes a person—of what bits of data are most important to determine who someone is—and the major players on the Web have quite different ways of approaching the problem.

Google’s filtering systems, for example, rely heavily on Web history and what you click on (click signals) to infer what you like and dislike. These clicks often happen in an entirely private context: The assumption is that searches for “intestinal gas” and celebrity gossip Web sites are between you and your browser. You might behave differently if you thought other people were going to see your searches. But it’s that behavior that determines what content you see in Google News, what ads Google displays—what determines, in other words, Google’s theory of you.

The basis for Facebook’s personalization is entirely different. While Facebook undoubtedly tracks clicks, its primary way of thinking about your identity is to look at what you share and with whom you interact. That’s a whole different kettle of data from Google’s: There are plenty of prurient, vain, and embarrassing things we click on that we’d be reluctant to share with all of our friends in a status update. And the reverse is true, too. I’ll cop to sometimes sharing links I’ve barely read—the long investigative piece on the reconstruction of Haiti, the bold political headline—because I like the way it makes me appear to others. The Google self and the Facebook self, in other words, are pretty different people. There’s a big difference between “you are what you click” and “you are what you share.”

Both ways of thinking have their benefits and drawbacks. With Google’s click-based self, the gay teenager who hasn’t come out to his parents can still get a personalized Google News feed with pieces from the broader gay community that affirm that he’s not alone. But by the same token, a self built on clicks will tend to draw us even more toward the items we’re predisposed to look at already—toward our most Pavlovian selves. Your perusal of an article on TMZ.com is filed away, and the next time you’re looking at the news, Brad Pitt’s marriage drama is more likely to flash on to the screen. (If Google didn’t persistently downplay porn, the problem would presumably be far worse.)

Facebook’s share-based self is more aspirational: Facebook takes you more at your word, presenting you as you’d like to be seen by others. Your Facebook self is more of a performance, less of a behaviorist black box, and ultimately it may be more prosocial than the bundle of signals Google tracks. But the Facebook approach has its downsides as well—to the extent that Facebook draws on the more public self, it necessarily has less room for private interests and concerns. The same closeted gay teenager’s information environment on Facebook might diverge more from his real self. The Facebook portrait remains incomplete.

Both are pretty poor representations of who we are, in part because there is no one set of data that describes who we are. “Information about our property, our professions, our purchases, our finances, and our medical history does not tell the whole story,” writes privacy expert Daniel Solove. “We are more than the bits of data we give off as we go about our lives.”

Digital animators and robotics engineers frequently run into a problem known as the uncanny valley. The uncanny valley is the place where something is lifelike but not convincingly alive, and it gives people the creeps. It’s part of why digital animation of real people still hasn’t hit the big screens: When an image looks almost like a real person, but not quite, it’s unsettling on a basic psychological level. We’re now in the uncanny valley of personalization. The doppelgänger selves reflected in our media are a lot like, but not exactly, ourselves. And as we’ll see, there are some important things that are lost in the gap between the data and reality.

To start with, Zuckerberg’s statement that we have “one identity” simply isn’t true. Psychologists have a name for this fallacy: fundamental attribution error. We tend to attribute peoples’ behavior to their inner traits and personality rather than to the situations they’re placed in. Even in situations where the context clearly plays a major role, we find it hard to separate how someone behaves from who she is.

And to a striking degree, our characteristics are fluid. Someone who’s aggressive at work may be a doormat at home. Someone who’s gregarious when happy may be introverted when stressed. Even some of our closest-held traits—our disinclination to do harm, for example—can be shaped by context. Groundbreaking psychologist Stanley Milgram demonstrated this when, in an oft-cited experiment at Yale in the 1960s, he got decent ordinary people to apparently electrocute other subjects when given the nod by a man in a lab coat.

There is a reason that we act this way: The personality traits that serve us well when we’re at dinner with our family might get in the way when we’re in a dispute with a passenger on the train or trying to finish a report at work. The plasticity of the self allows for social situations that would be impossible or intolerable if we always behaved exactly the same way. Advertisers have understood this phenomenon for a long time. In the jargon, it’s called day-parting, and it’s the reason that you don’t hear many beer ads as you’re driving to work in the morning. People have different needs and aspirations at eight A.M. than they do at eight P.M. By the same token, billboards in the nightlife district promote different products than billboards in the residential neighborhoods the same partiers go home to.

On his own Facebook page, Zuckerberg lists “transparency” as one of his top Likes. But there’s a downside to perfect transparency: One of the most important uses of privacy is to manage and maintain the separations and distinctions among our different selves. With only one identity, you lose the nuances that make for a good personalized fit.

Personalization doesn’t capture the balance between your work self and your play self, and it can also mess with the tension between your aspirational and your current self. How we behave is a balancing act between our future and present selves. In the future, we want to be fit, but in the present, we want the candy bar. In the future, we want to be a well-rounded, well-informed intellectual virtuoso, but right now we want to watch Jersey Shore. Behavioral economists call this present bias—the gap between your preferences for your future self and your preferences in the current moment.

The phenomenon explains why there are so many movies in your Netflix queue. When researchers at Harvard and the Analyst Institute looked at people’s movie-rental patterns, they were able to watch as people’s future aspirations played against their current desires. “Should” movies like An Inconvenient Truth or Schindler’s List were often added to the queue, but there they languished while watchers gobbled up “want” movies like Sleepless in Seattle. And when they had to choose three movies to watch instantly, they were less likely to choose “should” movies at all. Apparently there are some movies we’d always rather watch tomorrow.

At its best, media help mitigate present bias, mixing “should” stories with “want” stories and encouraging us to dig into the difficult but rewarding work of understanding complex problems. But the filter bubble tends to do the opposite: Because it’s our present self that’s doing all the clicking, the set of preferences it reflects is necessarily more “want” than “should.”

The one-identity problem isn’t a fundamental flaw. It’s more of a bug: Because Zuckerberg thinks you have one identity and you don’t, Facebook will do a worse job of personalizing your information environment. As John Battelle told me, “We’re so far away from the nuances of what it means to be human, as reflected in the nuances of the technology.” Given enough data and enough programmers, the context problem is solvable—and according to personalization engineer Jonathan McPhie, Google is working on it. We’ve seen the pendulum swing from the anonymity of the early Internet to the one-identity view currently in vogue; the future may look like something in between.

But the one-identity problem illustrates one of the dangers of turning over your most personal details to companies who have a skewed view of what identity is. Maintaining separate identity zones is a ritual that helps us deal with the demands of different roles and communities. And something’s lost when, at the end of the day, everything inside your filter bubble looks roughly the same. Your bacchanalian self comes knocking at work; your work anxieties plague you on a night out.

And when we’re aware that everything we do enters a permanent, pervasive online record, another problem emerges: The knowledge that what we do affects what we see and how companies see us can create a chilling effect. Genetic privacy expert Mark Rothstein describes how lax regulations around genetic data can actually reduce the number of people willing to be tested for certain diseases: If you might be discriminated against or denied insurance for having a gene linked to Parkinson’s disease, it’s not unreasonable just to skip the test and the “toxic knowledge” that might result.

In the same way, when our online actions are tallied and added to a record that companies use to make decisions, we might decide to be more cautious in our surfing. If we knew (or even suspected, for that matter) that purchasers of 101 Ways to Fix Your Credit Score tend to get offered lower-premium credit cards, we’d avoid buying the book. “If we thought that our every word and deed were public,” writes law professor Charles Fried, “fear of disapproval or more tangible retaliation might keep us from doing or saying things which we would do or say could we be sure of keeping them to ourselves.” As Google expert Siva Vaidhyanathan points out, “F. Scott Fitzgerald’s enigmatic Jay Gatsby could not exist today. The digital ghost of Jay Gatz would follow him everywhere.”

In theory, the one-identity, context-blind problem isn’t impossible to fix. Personalizers will undoubtedly get better at sensing context. They might even be able to better balance long-term and short-term interests. But when they do—when they are able to accurately gauge the workings of your psyche—things get even weirder.

Targeting Your Weak Spots

The logic of the filter bubble today is still fairly rudimentary: People who bought the Iron Man DVD are likely to buy Iron Man II; people who enjoy cookbooks will probably be interested in cookware. But for Dean Eckles, a doctoral student at Stanford and an adviser to Facebook, these simple recommendations are just the beginning. Eckles is interested in means, not ends: He cares less about what types of products you like than which kinds of arguments might cause you to choose one over another.

Eckles noticed that when buying products—say, a digital camera—different people respond to different pitches. Some people feel comforted by the fact that an expert or product review site will vouch for the camera. Others prefer to go with the product that’s most popular, or a money-saving deal, or a brand that they know and trust. Some people prefer what Eckles calls “high cognition” arguments—smart, subtle points that require some thinking to get. Others respond better to being hit over the head with a simple message.

And while most of us have preferred styles of argument and validation, there are also types of arguments that really turn us off. Some people rush for a deal; others think that the deal means the merchandise is subpar. Just by eliminating the persuasion styles that rub people the wrong way, Eckles found he could increase the effectiveness of marketing materials by 30 to 40 percent.

While it’s hard to “jump categories” in products—what clothing you prefer is only slightly related to what books you enjoy—“persuasion profiling” suggests that the kinds of arguments you respond to are highly transferrable from one domain to another. A person who responds to a “get 20% off if you buy NOW” deal for a trip to Bermuda is much more likely than someone who doesn’t to respond to a similar deal for, say, a new laptop.

If Eckles is right—and research so far appears to be validating his theory—your “persuasion profile” would have a pretty significant financial value. It’s one thing to know how to pitch products to you in a specific domain; it’s another to be able to improve the hit rate anywhere you go. And once a company like Amazon has figured out your profile by offering you different kinds of deals over time and seeing which ones you responded to, there’s no reason it couldn’t then sell that information to other companies. (The field is so new that it’s not clear if there’s a correlation between persuasion styles and demographic traits, but obviously that could be a shortcut as well.)

There’s plenty of good that could emerge from persuasion profiling, Eckles believes. He points to DirectLife, a wearable coaching device by Philips that figures out which arguments get people eating more healthily and exercising more regularly. But he told me he’s troubled by some of the possibilities. Knowing what kinds of appeals specific people respond to gives you power to manipulate them on an individual basis.

With new methods of “sentiment analysis, it’s now possible to guess what mood someone is in. People use substantially more positive words when they’re feeling up; by analyzing enough of your text messages, Facebook posts, and e-mails, it’s possible to tell good days from bad ones, sober messages from drunk ones (lots of typos, for a start). At best, this can be used to provide content that’s suited to your mood: On an awful day in the near future, Pandora might know to preload Pretty Hate Machine for you when you arrive. But it can also be used to take advantage of your psychology.

Consider the implications, for example, of knowing that particular customers compulsively buy things when stressed or when they’re feeling bad about themselves, or even when they’re a bit tipsy. If persuasion profiling makes it possible for a coaching device to shout “you can do it” to people who like positive reinforcement, in theory it could also enable politicians to make appeals based on each voter’s targeted fears and weak spots.

Infomercials aren’t shown in the middle of the night only because airtime then is cheap. In the wee hours, most people are especially suggestible. They’ll spring for the slicer-dicer that they’d never purchase in the light of day. But the three A.M. rule is a rough one—presumably, there are times in all of our daily lives when we’re especially inclined to purchase whatever’s put in front of us. The same data that provides personalized content can be used to allow marketers to find and manipulate your personal weak spots. And this isn’t a hypothetical possibility: Privacy researcher Pam Dixon discovered that a data company called PK List Management offers a list of customers titled “Free to Me—Impulse Buyers”; those listed are described as being highly susceptible to pitches framed as sweepstakes.

If personalized persuasion works for products, it can also work for ideas. There are undoubtedly times and places and styles of argument that make us more susceptible to believe what we’re told. Subliminal messaging is illegal because we recognize there are some ways of making an argument that are essentially cheating; priming people with subconsciously flashed words to sell them things isn’t a fair game. But it’s not such a stretch to imagine political campaigns targeting voters at times when they can circumvent our more reasonable impulses.

We intuitively understand the power in revealing our deep motivations and desires and how we work, which is why most of us only do that in day-to-day life with people whom we really trust. There’s a symmetry to it: You know your friends about as well as they know you. Persuasion profiling, on the other hand, can be done invisibly—you need not have any knowledge that this data is being collected from you—and therefore it’s asymmetrical. And unlike some forms of profiling that take place in plain sight (like Netflix), persuasion profiling is handicapped when it’s revealed. It’s just not the same to hear an automated coach say “You’re doing a great job! I’m telling you that because you respond well to encouragement!”

So you don’t necessarily see the persuasion profile being made. You don’t see it being used to influence your behavior. And the companies we’re turning over this data to have no legal obligation to keep it to themselves. In the wrong hands, persuasion profiling gives companies the ability to circumvent your rational decision making, tap into your psychology, and draw out your compulsions. Understand someone’s identity, and you’re better equipped to influence what he or she does.

A Deep and Narrow Path

Someday soon, Google Vice President Marissa Mayer says, the company hopes to make the search box obsolete. “The next step of search is doing this automatically,” Eric Schmidt said in 2010. “When I walk down the street, I want my smartphone to be doing searches constantly—‘did you know?’ ‘did you know?’ ‘did you know?’ ‘did you know?’” In other words, your phone should figure out what you would like to be searching for before you do.

In the fast-approaching age of search without search, identity drives media. But the personalizers haven’t fully grappled with a parallel fact: Media also shapes identity. Political scientist Shanto Iyengar calls one of primary factors accessibility bias, and in a paper titled “Experimental Demonstrations of the ‘Not-So-Minimal’ Consequences of Television News,’” in 1982, he demonstrated how powerful the bias is. Over six days, Iyengar asked groups of New Haven residents to watch episodes of a TV news program, which he had doctored to include different segments for each group.

Afterward, Iyengar asked subjects to rank how important issues like pollution, inflation, and defense were to them. The shifts from the surveys they’d filled out before the study were dramatic: “Participants exposed to a steady stream of news about defense or about pollution came to believe that defense or pollution were more consequential problems,” Iyengar wrote. Among the group that saw the clips on pollution, the issue moved from fifth out of six in priority to second.

Drew Westen, a neuropsychologist whose focus is on political persuasion, demonstrates the strength of this priming effect by asking a group of people to memorize a list of words that include moon and ocean. A few minutes later, he changes topics and asks the group which detergent they prefer. Though he hasn’t mentioned the word, the group’s show of hands indicates a strong preference for Tide.

Priming isn’t the only way media shape our identities. We’re also more inclined to believe what we’ve heard before. In a 1977 study by Hasher and Goldstein, participants were asked to read sixty statements and mark whether they were true or false. All of the statements were plausible, but some of them (“French horn players get cash bonuses to stay in the Army”) were true; others (“Divorce is only found in technically advanced societies”) weren’t. Two weeks later, they returned and rated a second batch of statements in which some of the items from the first list had been repeated. By the third time, two weeks after that, the subjects were far more likely to believe the repeated statements. With information as with food, we are what we consume.

All of these are basic psychological mechanisms. But combine them with personalized media, and troubling things start to happen. Your identity shapes your media, and your media then shapes what you believe and what you care about. You click on a link, which signals an interest in something, which means you’re more likely to see articles about that topic in the future, which in turn prime the topic for you. You become trapped in a you loop, and if your identity is misrepresented, strange patterns begin to emerge, like reverb from an amplifier.

If you’re a Facebook user, you’ve probably run into this problem. You look up your old college girlfriend Sally, mildly curious to see what she is up to after all these years. Facebook interprets this as a sign that you’re interested in Sally, and all of a sudden her life is all over your news feed. You’re still mildly curious, so you click through on the new photos she’s posted of her kids and husband and pets, confirming Facebook’s hunch. From Facebook’s perspective, it looks as though you have a relationship with this person, even if you haven’t communicated in years. For months afterward, Sally’s life is far more prominent than your actual relationship would indicate. She’s a “local maximum”: Though there are people whose posts you’re far more interested in, it’s her posts that you see.

In part, this feedback effect is due to what early Facebook employee and venture capitalist Matt Cohler calls the local-maximum problem. Cohler was an early employee at Facebook, and he’s widely considered one of Silicon Valley’s smartest thinkers on the social Web.

The local-maximum problem, he explains to me, shows up any time you’re trying to optimize something. Say you’re trying to write a simple set of instructions to help a blind person who’s lost in the Sierra Nevadas find his way to the highest peak. “Feel around you to see if you’re surrounded by downward-sloping land,” you say. “If you’re not, move in a direction that’s higher, and repeat.”

Programmers face problems like this all the time. What link is the best result for the search term “fish”? Which picture can Facebook show you to increase the likelihood that you’ll start a photo-surfing binge? The directions sound pretty obvious—you just tweak and tune in one direction or another until you’re in the sweet spot. But there’s a problem with these hill-climbing instructions: They’re as likely to end you up in the foothills—the local maximum—as they are to guide you to the apex of Mount Whitney.

This isn’t exactly harmful, but in the filter bubble, the same phenomenon can happen with any person or topic. I find it hard not to click on articles about gadgets, though I don’t actually think they’re that important. Personalized filters play to the most compulsive parts of you, creating “compulsive media” to get you to click things more. The technology mostly can’t distinguish compulsion from general interest—and if you’re generating page views that can be sold to advertisers, it might not care.

The faster the system learns from you, the more likely it is that you can get trapped in a kind of identity cascade, in which a small initial action—clicking on a link about gardening or anarchy or Ozzy Osbourne—indicates that you’re a person who likes those kinds of things. This in turn supplies you with more information on the topic, which you’re more inclined to click on because the topic has now been primed for you.

Especially once the second click has occurred, your brain is in on the act as well. Our brains act to reduce cognitive dissonance in a strange but compelling kind of unlogic—“Why would I have done x if I weren’t a person who does x—therefore I must be a person who does x.”Each click you take in this loop is another action to self-justify—“Boy, I guess I just really love ‘Crazy Train.’” When you use a recursive process that feeds on itself, Cohler tells me, “You’re going to end up down a deep and narrow path.” The reverb drowns out the tune. If identity loops aren’t counteracted through randomness and serendipity, you could end up stuck in the foothills of your identity, far away from the high peaks in the distance.

And that’s when these loops are relatively benign. Sometimes they’re not.

We know what happens when teachers think students are dumb: They get dumber. In an experiment done before the advent of ethics boards, teachers were given test results that supposedly indicated the IQ and aptitude of students entering their classes. They weren’t told, however, that the results had been randomly redistributed among students. After a year, the students who the teachers had been told were bright made big gains in IQ. The students who the teachers had been told were below average had no such improvement.

So what happens when the Internet thinks you’re dumb? Personalization based on perceived IQ isn’t such a far-fetched scenario—Google Docs even offers a helpful tool for automatically checking the grade-level of written text. If your education level isn’t already available through a tool like Acxiom, it’s easy enough for anyone with access to a few e-mails or Facebook posts to infer. Users whose writing indicates college-level literacy might see more articles from the New Yorker; users with only basic writing skills might see more from the New York Post.

In a broadcast world, everyone is expected to read or process information at about the same level. In the filter bubble, there’s no need for that expectation. On one hand, this could be great—vast groups of people who have given up on reading because the newspaper goes over their heads may finally connect with written content. But without pressure to improve, it’s also possible to get stuck in a grade-three world for a long time.

Incidents and Adventures

In some cases, letting algorithms make decisions about what we see and what opportunities we’re offered gives us fairer results. A computer can be made blind to race and gender in ways that humans usually can’t. But that’s only if the relevant algorithms are designed with care and acuteness. Otherwise, they’re likely to simply reflect the social mores of the culture they’re processing—a regression to the social norm.

In some cases, algorithmic sorting based on personal data can be even more discriminatory than people would be. For example, software that helps companies sift through résumés for talent might “learn” by looking at which of its recommended employees are actually hired. If nine white candidates in a row are chosen, it might determine that the company isn’t interested in hiring black people and exclude them from future searches. “In many ways,” writes NYU sociologist Dalton Conley, “such network-based categorizations are more insidious than the hackneyed groupings based on race, class, gender, religion, or any other demographic characteristic.” Among programmers, this kind of error has a name. It’s called overfitting.

The online movie rental Web site Netflix is powered by an algorithm called CineMatch. To start, it was pretty simple. If I had rented the first movie in the Lord of the Rings trilogy, let’s say, Netflix could look up what other movies Lord of the Rings watchers had rented. If many of them had rented Star Wars, it’d be highly likely that I would want to rent it, too.

This technique is called kNN (k-nearest-neighbor), and using it CineMatch got pretty good at figuring out what movies people wanted to watch based on what movies they’d rented and how many stars (out of five) they’d given the movies they’d seen. By 2006, CineMatch could predict within one star how much a given user would like any movie from Netflix’s vast hundred-thousand-film emporium. Already CineMatch was better at making recommendations than most humans. A human video clerk would never think to suggest Silence of the Lambs to a fan of The Wizard of Oz, but CineMatch knew people who liked one usually liked the other.

But Reed Hastings, Netflix’s CEO, wasn’t satisfied. “Right now, we’re driving the Model-T version of what’s possible,” he told a reporter in 2006. On October 2, 2006, an announcement went up on the Netflix Web site: “We’re interested, to the tune of $1 million.” Netflix had posted an enormous swath of data—reviews, rental records, and other information from its user database, scrubbed of anything that would obviously identify a specific user. And now the company was willing to give $1 million to the person or team who beat CineMatch by more than 10 percent. Like the longitude prize, the Netflix Challenge was open to everyone. “All you need is a PC and some great insight,” Hastings declared in the New York Times.

After nine months, about eighteen thousand teams from more than 150 countries were competing, using ideas from machine learning, neural networks, collaborative filtering, and data mining. Usually, contestants in high-stakes contests operate in secret. But Netflix encouraged the competing groups to communicate with one another and built a message board where they could coordinate around common obstacles. Read through the message board, and you get a visceral sense of the challenges that bedeviled the contestants during the three-year quest for a better algorithm. Overfitting comes up again and again.

There are two challenges in building pattern-finding algorithms. One is finding the patterns that are there in all the noise. The other problem is the opposite: not finding patterns in the data that aren’t actually really there. The pattern that describes “1, 2, 3” could be “add one to the previous number” or “list positive prime numbers from smallest to biggest.” You don’t know for sure until you get more data. And if you leap to conclusions, you’re overfitting.

Where movies are concerned, the dangers of overfitting are relatively small—many analog movie watchers have been led to believe that because they liked The Godfather and The Godfather: Part II, they’ll like The Godfather: Part III. But the overfitting problem gets to one of the central, irreducible problems of the filter bubble: Overfitting and stereotyping are synonyms.

The term stereotyping (which in this sense comes from Walter Lippmann, incidentally) is often used to refer to malicious xenophobic patterns that aren’t true—“people of this skin color are less intelligent” is a classic example. But stereotypes and the negative consequences that flow from them aren’t fair to specific people even if they’re generally pretty accurate.

Marketers are already exploring the gray area between what can be predicted and what predictions are fair. According to Charlie Stryker, an old hand in the behavioral targeting industry who spoke at the Social Graph Symposium, the U.S. Army has had terrific success using social-graph data to recruit for the military—after all, if six of your Facebook buddies have enlisted, it’s likely that you would consider doing so too. Drawing inferences based on what people like you or people linked to you do is pretty good business. And it’s not just the army. Banks are beginning to use social data to decide to whom to offer loans: If your friends don’t pay on time, it’s likely that you’ll be a deadbeat too. “A decision is going to be made on creditworthiness based on the creditworthiness of your friends,” Stryker said. “There are applications of this technology that can be very powerful,” another social targeting entrepreneur told the Wall Street Journal. “Who knows how far we’d take it?”

Part of what’s troubling about this world is that companies aren’t required to explain on what basis they’re making these decisions. And as a result, you can get judged without knowing it and without being able to appeal. For example, LinkedIn, the social job-hunting site, offers a career trajectory prediction site; by comparing your résumé to other peoples’ who are in your field but further along, LinkedIn can forecast where you’ll be in five years. Engineers at the company hope that soon it’ll be able to pinpoint career choices that lead to better outcomes—“mid-level IT professionals like you who attended Wharton business school made $25,000/year more than those who didn’t.” As a service to customers, it’s pretty useful. But imagine if LinkedIn provided that data to corporate clients to help them weed out people who are forecast to be losers. Because that could happen entirely without your knowledge, you’d never get the chance to argue, to prove the prediction wrong, to have the benefit of the doubt.

If it seems unfair for banks to discriminate against you because your high school buddy is bad at paying his bills or because you like something that a lot of loan defaulters also like, well, it is. And it points to a basic problem with induction, the logical method by which algorithms use data to make predictions.

Philosophers have been wrestling with this problem since long before there were computers to induce with. While you can prove the truth of a mathematical proof by arguing it out from first principles, the philosopher David Hume pointed out in 1772 that reality doesn’t work that way. As the investment cliché has it, past performance is not indicative of future results.

This raises some big questions for science, which is at its core a method for using data to predict the future. Karl Popper, one of the preeminent philosophers of science, made it his life’s mission to try to sort out the problem of induction, as it came to be known. While the optimistic thinkers of the late 1800s looked at the history of science and saw a journey toward truth, Popper preferred to focus on the wreckage along the side of the road—the abundance of failed theories and ideas that were perfectly consistent with the scientific method and yet horribly wrong. After all, the Ptolemaic universe, with the earth in the center and the sun and planets revolving around it, survived an awful lot of mathematical scrutiny and scientific observation.

Popper posed his problem in a slightly different way: Just because you’ve only ever seen white swans doesn’t mean that all swans are white. What you have to look for is the black swan, the counterexample that proves the theory wrong. “Falsifiability,” Popper argued, was the key to the search for truth: The purpose of science, for Popper, was to advance the biggest claims for which one could not find any countervailing examples, any black swans. Underlying Popper’s view was a deep humility about scientifically induced knowledge—a sense that we’re wrong as often as we’re right, and we usually don’t know when we are.

It’s this humility that many algorithmic prediction methods fail to build in. Sure, they encounter people or behaviors that don’t fit the mold from time to time, but these aberrations don’t fundamentally compromise their algorithms. After all, the advertisers whose money drives these systems don’t need the models to be perfect. They’re most interested in hitting demographics, not complex human beings.

When you model the weather and predict there’s a 70 percent chance of rain, it doesn’t affect the rain clouds. It either rains or it doesn’t. But when you predict that because my friends are untrustworthy, there’s a 70 percent chance that I’ll default on my loan, there are consequences if you get me wrong. You’re discriminating.

The best way to avoid overfitting, as Popper suggests, is to try to prove the model wrong and to build algorithms that give the benefit of the doubt. If Netflix shows me a romantic comedy and I like it, it’ll show me another one and begin to think of me as a romantic-comedy lover. But if it wants to get a good picture of who I really am, it should be constantly testing the hypothesis by showing me Blade Runner in an attempt to prove it wrong. Otherwise, I end up caught in a local maximum populated by Hugh Grant and Julia Roberts.

The statistical models that make up the filter bubble write off the outliers. But in human life it’s the outliers who make things interesting and give us inspiration. And it’s the outliers who are the first signs of change.

One of the best critiques of algorithmic prediction comes, remarkably, from the late-nineteenth-century Russian novelist Fyodor Dostoyevsky, whose Notes from Underground was a passionate critique of the utopian scientific rationalism of the day. Dostoyevsky looked at the regimented, ordered human life that science promised and predicted a banal future. “All human actions,” the novel’s unnamed narrator grumbles, “will then, of course, be tabulated according to these laws, mathematically, like tables of logarithms up to 108,000, and entered in an index… in which everything will be so clearly calculated and explained that there will be no more incidents or adventures in the world.”

The world often follows predictable rules and falls into predictable patterns: Tides rise and fall, eclipses approach and pass; even the weather is more and more predictable. But when this way of thinking is applied to human behavior, it can be dangerous, for the simple reason that our best moments are often the most unpredictable ones. An entirely predictable life isn’t worth living. But algorithmic induction can lead to a kind of information determinism, in which our past clickstreams entirely decide our future. If we don’t erase our Web histories, in other words, we may be doomed to repeat them.