This page has been migrated from an earlier version of this site. Links and images may be broken.
Before leaving for a holiday (it was lovely; thanks for asking) I was going through a trio of books on the topical topic of “prediction is difficult, especially of the future”. I decreed that Dan Gardner’s Future Babble was limited, but otherwise OK, and then deemed Tim Harford’s Adapt a failed attempt to justify free-market thinking in the aftermath of its biggest failure in decades.
So now it’s onto the final book of the trio, and my favourite by some distance. Australian Duncan Watts is a physicist-turned sociologist who now works! at! Yahoo! (I called him American a few posts ago. Thanks to Kevin Horgan for correcting me.) You may know Watts from such hits as his explication of network science Six Degrees of Freedom; his Music Lab experiments with Matthew Salganik and Peter Dodds (pdf) showing that social influence can overwhelm any special quality of particular songs in separating hits from misses; and his arguments against Malcolm Gladwell’s The Tipping Point. Each of these reappears in his latest book, Everything is Obvious (Once You Know the Answer) - henceforth EIO (home page).
(Attention conservation notice: for some reason this took for ever to write and I still don’t like it, but I said I’d post it so I’m damn well going to, and then I can move on.)
If I were writing for a real publication I’d try to be dispassionate and objective, but the fact is that EIO’s goals are ones that I am hugely sympathetic to, and so I found myself cheering Watts along rather than reading him critically. I try not to do this, but between you and me and the wall we all do it to some extent. Certainly the reviewers of Adapt could have used a little less enthusiasm and a little more critical thinking before putting their praise in print. Still, I do think that EIO is the book that Adapt and Future Babble were trying to be.
According to its subtitle, EIO is about “how common sense fails us”, but this is misleading. “Common sense” has become a shorthand for anti-intellectual conservative approaches to government and social problems based on down-home practical experience, as in Common Sense Revolution, but Watts has a more ambitious target than that; “common sense” is the need (among intellectuals and others alike) to impose patterns on current events and on history, and our belief that once we see these patterns we will have a better grasp of the future.
On the one hand, the achievements of common sense are easily overlooked, but remarkable. Our socially-honed and culturally-specific intuitions are remarkably good at helping us to navigate the maze of implicit rules and norms that help society function. Common sense has proven subtle enough to derail whole avenues of artificial intelligence research.
On the other hand, the strengths of common sense in navigating the particulars of travelling the subway or getting by in the workplace are weaknesses when it comes to understanding anything other than everyday life. Common sense is “not so much a worldview as a grab bag of logically inconsistent, often contradictory beliefs, each of which seems right at the time but carries no guarantee of being right at any other time” (17). When it comes to understanding society, we appeal to it at our peril.
Watts’s most obvious target is the kind of thinking that is commonplace in business, in politics, and in punditry – analogies, stories, and overextended but half-baked theories. But as a sociologist, Watts also has in his sights the economists’ belief that sociology is plagued by woolly thinking, and that economists can do sociologists’ work more rigorously and with more insight than sociologists themselves because economists understand incentives. He quotes from Freakonomics: “The typical economist believes that the world has not yet invented a problem that he [sic] cannot fix if given a free hand to design the proper incentive scheme” (51).
The usual modern take against homo economicus theories of society is that we are not rational beings. But the problem is not just that “our mental model of individual behavior is systematically flawed”, although Watts does run through failures of framing, priming and other foibles of the Predictably Irrational kind. The problem I’ve always had with such psychological explanations of our failures is that, once we are made aware of our systematic flaws, we can surely work on overcoming them. Would that solve the problem?
Well no. And it’s not just that we sumble into fallacies of composition either, although he does show why intuitions and theories based on “representative individuals” whose “actions stand in for the actions and interactions of the many” are both tempting and doomed to fail. The focus on “special individuals” as the sparks that light the fires of viral success, Malcolm Gladwell’s “Law of the Few”, also comes in for debunking, including some convincing experiments based on Twitter cascades. And expertise as a whole is not of much use: Chapter 6 covers much of what’s in Future Babble. But the smartest and most self-confident will still see themselves, plausibly, as able to surmount these barriers, cutting through the deadwood to find real causes, real incentives, and real mechanisms.
Watts continues though. It is in his treatment of successes and failures that he makes his strongest arguments. Watts convincingly shows many suggested explanations of successes, from the Mona Lisa to Facebook, to be exercises in circular thinking, where “Harry Potter was successful because it had exactly the attributes of Harry Potter, and not something else” (60). It is tempting to think that there is something special about the Mona Lisa that causes it to be the most famous painting in the world, some quality that it has that no other painting has. But there isn’t.
In the Music Lab experiments, different “worlds” of listeners were given a set of songs to listen to and, in some cases, information on the listening history of others in their world. The experiment showed that songs that were hits in some worlds were unpopular in others, and that this variation came about specifically because of the information about others’ listening histories. Not only is it a mistake to look for Gladwell-like “special people”, it’s also a mistake to look for special qualities in hit songs as the “cause” of their success. As the fictional Mark Zuckerberg says to the Winklevoss twins in The Social Network, “if you had invented Facebook, you would have invented Facebook”. There was no crucial idea that separated Facebook from the pack; it just separated from the pack because social networks are governed by cumulative advantage, and that’s how cumulative advantage works.
And he doesn’t stop there. We’ve perhaps read before that “history is only run once”, causing us to mistakenly “perceive what actually happened as having been inevitable” (112), but Watts goes further in explaining why we misunderstand the causes of events: events themselves are identifiable only in hindsight. Was “the US financial crisis of 2008” a blip in the recent history of capitalism, or was it just the first part of “the international financial crisis of 2008 to 2015”, including the splitup of the euro zone and who knows what else? We can only know in hindsight.
In summary, Watts is saying not only that the “right lever” is difficult to find when it comes to understanding society, but that in many social phenomena there never was a lever at all; “what appear to us to be causal explanations are in fact just stories — descriptions of what happened that tell us little, if anything, about the mechanisms at work” (27).
There’s a lot here, and by two-thirds of the way through the book I was happily convinced that I would never again need to reach for a business strategy or populist social science book. It’s a thorough and convincing debunking of attempts to understand the chaotic path of social progress.
The difficulty, of course, is that once the Emperor’s nudity has been pointed out, the man still needs a set of clothes. What to put in the place of mistaken theories? Part II is an attempt to address this question, and it’s less successful than part I. There are three main chapters (Chapter 7 belongs, to my mind, in Part I rather than Part II). Chapter 8 covers much the same ground as Adapt, with some of the same suggestions: experiment, measure and react, rely on prediction as little as possible. Condensed into one chapter I didn’t mind this as much as I did in Adapt, but it’s still the weakest chapter in the book. It’s a smattering of ideas, some stronger than others, but still searching for that magic recipe which he has just convinced us doesn’t exist.
Chapter 9 is far more interesting though. If success and failure are governed largely by luck, how does that affect our views of society? Watts looks at theories of justice in the light of what Part I has told us, and comes out firmly in favour of an egalitarian, or at least Rawlsian, point of view. It’s just one chapter, and Watts would (I would guess) be the first to admit it’s just an introduction, but it’s a bold step for a book on popular social science to take, and a valuable one.
It’s enough that I forgive Watts his final chapter, which is really a set of reflections on physics and sociology that somehow forgets to mention that of all the natural sciences to base social theories on, especially when you have convinced us that specifics of particular situations are crucial, physics is surely the least plausible. It’s easy to forget how little predictive success physics has. Sure it does a good job with the tides, but even the best physics theories of complex phenomena (say the BCS theory of superconductivity for example, or the Kondo model) have little success when it comes to prediction. BCS did not guide or anticipate the discovery of high-temperature superconductors, for example.
After reading these three books I am left with several unanswered questions about how to proceed once we know that large parts of the future are hidden from view? When it comes to decisions, well we still need to make them (including big and irreversible decisions) even in the face of uncertainty, but can we still put heart and soul into new initiatives when we know that their outcome is, at least in part, out of our control? How do you motivate teams without misrepresenting the role of luck? How do we approach questions of reward and punishment when the consequences of our actions are unforeseeable? I’m sure there’s lots out there on these questions, and I think I’ll be looking for it with a new appreciation of their importance. Any suggestions on places to start?