In March 2012, close observers of the Republican presidential primary process were mystified by some of Rick Santorum’s victories. The underdog candidate, known for his extreme social conservative politics, was not only doing relatively well compared to the much better-funded Mitt Romney, he was consistently performing better than polls predicted. In 11 heavily polled states, Santorum performed an average of 2.3% better than predicted by polling, even picking up unexpected victories in Alabama, Iowa, and Mississippi.

Puzzling over the results, Nate Silver took his New York Times-hosted blog, FiveThirtyEight, to crunch some numbers. To see whether other candidates were over- or under-performing, he produced a table with the average actual performance of each of the Republican candidates and their polling estimates in two different subsets of states. Silver hypothesized that the disparity might be explained by the underrepresentation of cell phone-only respondents who might skew toward Santorum. Considering this possibility, he presented a chart examining the correlation between Santorum’s performance and the percentage of cell phone-only polls. Unfortunately, the relationship was weak. Silver had to confess that he had no convincing answer to the mystery.

Still, the Santorum blog post illustrates Silver’s modus operandi. He’s been called a “statistical boy genius,” a “numbers wizard,” and a “hero to geeks” in his willingness to identify a question, seize on some data, and apply any and all statistical tools to answer it. He writes in straightforward, mercifully jargon-free prose, but like every good social scientist, Silver’s chief devotion is to the findings, not to any particular ideology. In the always-risky field of political prediction, this approach has proven remarkably successful. On the eve of the 2008 Presidential election, Silver correctly predicted 49 of 50 states and estimated the population vote within 0.4% of President Obama’s 6.1% margin of victory.Nate Silver has been called a “statistical boy genius” and a “hero to geeks.” He writes in straightforward, mercifully jargon-free prose, and his devotion is to the findings. This approach has proven remarkably successful.

And it’s not just politics. Silver made his name as one of the original MoneyBall–ers, developing the system of baseball player performance prediction known as PECOTA. More recently, he has created quality of life rankings for New York City neighborhoods, predicted Oscar and World Cup winners, and used online dating data to determine that Wednesday nights are optimal for “singles on the prowl.” Silver’s curiosity leads him to a wide variety of questions, and his process of answering those questions invariably uses statistical methods, traditionally the province of social scientists.

With his rapid-response blogging (he even live-tweeted the NBA Finals), Silver is clearly a man of his age. His versatile public intellectualism is hardly new. But compared to public intellectuals of the past, who benefited from widespread readership of serious nonfiction, Silver writes for a far narrower segment of highly engaged elites with relatively rarefied knowledge. As the American public intellectually disengages with civic life, the language of political discourse has become technocratic. To be sure, Silver is a democratizer of knowledge, but one whose public extends only as far as those with a semester of college stats under their belt. Today, to talk politics is to speak in statistics.Today, to talk politics is to speak in statistics.

The Changing Marketplace of Ideas

“Every year, 70 readers die and only 2 are replaced,” novelist Philip Roth told journalist David Remnick in 2000. Roth’s word choice was deliberate. In an age of e-mails, tweets, Facebook posts, and text messages, it’s impossible to argue people aren’t reading. What was and is fast becoming extinct is the reader, devoted to consuming and reflecting upon serious books. “Literature takes a habit of mind that has disappeared. It requires silence, some form of isolation, and sustained concentration in the presence of an enigmatic thing,” said Roth. Despite its many benefits for mental health and civil society, far fewer people are reading books than in the past. So, though Roth’s invented numbers are hyperbolic, according to the National Endowment for the Arts, the percentage of American adults reading any kind of book without being assigned to do so declined from 67% in 1982 to 54% in 2008 (the most recent year of NEA’s study). And if literature is in bad shape, the market for serious nonfiction is even worse. With the exception of a Freakonomics here or a SuperFreakonomics there, few Americans now read nonfiction informed by social scientific research or even rigorous journalism. While we must avoid the tendency to romanticize a great age of high-minded literacy that never quite was, it seems clear the audience for public intellectualism has shrunk.

At the same time, several other factors have converged to transform the nature of political discourse. As powerful computers have become widely available, social scientists have developed increasingly complex statistical methods, capable of analyzing dozens of models and tens of thousands of cases in seconds. This democratized access to computers has also helped statistics flourish in the same way it’s spurred creativity in software development. At the same time 12-year-old Mark Zuckerberg, future founder of Facebook, programmed video games in his parents’ house in White Plains, 18-year-old Silver was running statistics software to beat his high school buddies in fantasy baseball.

Combined with expanded access to higher education, these factors contributed to the creation of a new technocratic elite. The segment of the public now interested in reading serious political analysis is more educated, more statistically literate, and more homogeneous. The marketplace of ideas has shrunk, and it now trades in more specialized goods.The marketplace of ideas has shrunk, and it now trades in more specialized goods.

According to the Archives of Internal Medicine in 2010, just 24% of Americans can express 1 in 1000 as a percentage. The Annie E. Casey Foundation revealed in 2002 that only 30% can correctly identify what “margin of error” means given four multiple choice options. Yet Nate Silver regularly posts outputs from multivariate regression analyses, resplendent with unstandardized coefficients, standard errors, and R2s. These methods may seem like child’s play for academics, but the percentage of the American population who can interpret such knowledge is minuscule. Even with his immense skill in translating statistics for the public, Silver merely expands his audience from the highly rarefied world of quantitative researchers to the only somewhat rarefied world of New York Times blog readers. And while the group that can grasp such specialized knowledge is small, Silver’s technical, social scientific analysis has made him the toast (and inspiration) of the political class. In 2008, for example, the Obama campaign hosted a statistics training camp for campaign volunteers and, in 2012, Obama’s reelection campaign hired a “data brigade” to produce predictive statistical models—the same type of analysis that changed baseball is changing politics.

If the content and the audience for public intellectualism has changed, so too has the medium. It’s practically cliché to note that the slow, meditative form of the book has been replaced by the short, rapid-response media of tweets and blog posts. Magazine articles qualify as “long form” analysis. Silver’s primary outlet is his blog, but he also offers immediate reactions to breaking events by Twitter and more extensive analyses in media like the New York Times Magazine.

Silver exemplifies this form of web-based, technocratic, public intellectualism. He’s often called a “genius” for his ability to find clever ways to inform public debates with some nifty bit of data analysis. But, as Malcolm Gladwell tells us in Outliers, the term “genius” begs suspicion. Being smart is necessary, but not sufficient. Gladwell offers the example of Bill Gates, who achieved his fortune in part because of intelligence and passion, but also because of a series of highly unusual opportunities. Gates was born into a privileged background that allowed him access to 10,000 hours with computers at precisely the right moment in time.

The same is true for Silver. Born in 1978, Silver attended the University of Chicago in the late 1990s, a huge growth period for advanced statistics (remember, this is when fast, powerful, but compact computers had become publicly available). As a post-grad, he was at the perfect age to be among the vanguard of bloggers. Finally, he benefited from a historically rare level of public interest in the 2008 Presidential election. Silver may have been the first to post Stata output on the New York Times’ web site, but if he hadn’t, someone else surely would have.

Make no mistake, Silver is a very smart guy with a nose for useful data. My point is that he represents a mode of intellectualism that has resulted from major changes in the content, medium, and audience for political analysis. Had Edward R. Murrow turned 30 in 2008, he, too, might have tweeted about predictive models.

Blogger Sociology

Statistics are a tool. Statistical analysis has, for example, led to the creation of new financial instruments on Wall Street and more hi-tech methods of crime detection in police departments around the country. Still, there’s much to bemoan about their immense influence in society, from their role in creating the exotic investments that led to the 2008 banking crisis to their potential to paint reductionist portraits of complex social circumstances. So, as we wouldn’t despise the pencil, only the contemptible writer who put it to paper, there’s nothing inherently wrong with statistical analysis. In Silver’s case, his evenhanded use of statistics has nearly always enriched public debates with new sociological insights. At his best, Silver has the publishing speed and readability of journalism with the systematic evidence and complexity of academics. Silver’s style indisputably reflects this freedom and great inventiveness in conjuring cleverly framed questions and elegant designs to answer them. His ethos might be: get an idea and a scrap of data. Publish the results in easily digested bites.

Unlike academics, Silver is unburdened by the constraining forces of peer review, turgid and esoteric disciplinary jargon, and the unwieldy format of academic manuscripts. He need not kowtow to past literature, offer exacting descriptions of his methods, or explain in tedious detail how his findings contribute to existing theory. The check on Silver’s methods is, instead, his visibility. All his posts are immediately scrutinized by highly educated readers. In 2011, he began to acknowledge and often address these critiques in a regular post called “Reads and Reactions.”

Unlike conventional journalists, he is not constrained by the “objectivity norm” or journalistic style. While most J-schools insist budding reporters keep themselves out of the story, one of the great pleasures of Silver’s work is his original analysis and perspective. Unafraid to admit political opinions, he establishes credibility with his commitment to empirical evidence. As a blogger, he can write his own style guide.

In these ways, Silver has developed his own sociology, exploring social divisions, challenging “common sense” assumptions with evidence, and critiquing social scientific methods. For example, one of Silver’s long-term projects (an interest shared by many academics) is to better understand the ways in which Americans are divided, particularly in political preferences. In a post-election think piece in Esquire, Silver wrote, “If Bill Clinton was the first black president, then Barack Obama might be the first urban one.” He went on to explain that, since at least the late-1980s, rural areas have favored Republicans and urban areas have tilted Democratic. But in 2008, Obama saw a 10.5 million vote margin of victory in urban areas, while McCain had a significant edge in rural areas. This difference, Silver explained, wasn’t simply a function of the disproportionate number of black and young voters in urban areas—white and older urban people disproportionately voted for Obama, too. With more people living in cities than in rural areas, Obama’s victory was virtually guaranteed. Instead, Obama won urban areas by such a lopsided margin in part because his identity was, Silver said, “unmistakably urban: pragmatic, superior, hip, stubborn, multicultural.” All the symbols that appealed to an urban voting public—Obama’s education, race, fluency with pop culture—represented a cultural threat to the orthodox worldview in rural areas. More importantly urban voters tend to lean Democratic and there were more urban voters than ever before.Silver has developed his own sociology, exploring social divisions, challenging “common sense” with evidence, and critiquing social scientific methods.

This fairly simple analysis of voting data allowed Nate Silver to catch an important voting cleavage many pundits missed. The results of the 2008 election weren’t determined by facts unique to this political race (that is, that black voters wanted a black president or young voters liked Jay-Z references). The results emerged from long-term demographic shifts.

For Silver, such divisions of race, class, gender, age, education, and urbanity are crucial to understanding the American political landscape. His phenomenal success in predicting the outcomes of the 2008 Democratic primary and, later, the general election were built on a statistical formula that modeled the demographic characteristics of various voting districts. In this way, he could tell that districts with older, more female voters would lean heavily toward Hillary Clinton. A model that combined factors of age, education, race, and gender with current polling, produced far better predictions than polling data alone.

Understanding the consequences of social divisions is essential to Silver’s brand of political sociology, but so is challenging conventional wisdom with empirical evidence. Among the most popular political assumptions is former Clinton adviser James Carville’s assertion “It’s the economy, stupid.” For Carville, this conviction stems from a gut-level response to his personal experiences on the campaign trail—elections are won because of economics. For Silver, this is a testable question.

In November 2011, Silver posted a blog analyzing which of 43 economic measures (from Consumer Price Index to Change in Nonfarm Payrolls) best predicts the popular vote in presidential elections. Contrary to Carville’s view, Silver found no economic measure could explain more than 46% of the popular vote in elections between 1948 and 2008. In response, for the 2012 presidential election, Silver has developed state-level models that combine polling averages, previous presidential election results, and state demographics. These results are aggregated into a national model and combined with economic measures. Using this model, which incorporates economic factors as well as social and political characteristics, Silver then runs 10,000 simulated election outcomes. As of mid-June 2012, Silver’s model showed Obama winning in 63% of these simulated elections. The end result of Silver’s work is increasing the complexity of political debate by overturning poor assumptions and offering sophisticated alternatives.

Silver’s skepticism over the predictive power of “economic fundamentals” also speaks to a final characteristic of his approach to statistical analysis of society. He is deeply humble about the limitations of prediction. Like all good social scientists, Silver tends to hammer away about these limitations, even as readers drool in anticipation of his next forecasting model. In March 2012, he warned “…we can get into trouble when we exaggerate how much we know about the future. Although election forecasting is a relatively obscure topic, you’ll see the same mistakes in fields like finance and earthquake prediction in which the stakes are much higher.”

The Dangers of Statistics

Despite the obvious value of statistics for advancing public debate, there are serious problems with this mode of public intellectualism. Silver’s methodology is inherently exclusive. He’s a brilliant translator of statistical ideas into more accessible terms, but statistics remain a form of elite knowledge. So, if the weakness of most journalistic coverage of politics and social issues is a lack of technical precision and sophistication, its core strength was its ability to connect with a broad audience. Silver’s model has the opposite problem: It excludes all but elite readers. Paul Krugman, a Nobel Prize-winning economist and Silver’s colleague at the New York Times, seems aware of this boundary. When one of his blog posts is fairly technical or statistical, Krugman appends the title with “(wonky).” In doing so, Krugman warns that all who tread here must come armed with a fluency in the language of statistics and economics.Despite the value of statistics for public debate, Silver’s method of public intellectualism is inherently exclusive. Statistics remain a form of elite knowledge.

Of course, readers need not follow such links, and they can certainly skip the confusing bits of Silver’s posts. When they do so, however, the gap in knowledge between elites and the rest of the public widens. Perhaps more tellingly, the bigger divide goes beyond those who do (and do not) read Krugman and Silver’s blogs—it’s the gap between between blog readers and the general readers of yesteryear, most of whom are no longer reading books, never mind statistically-oriented blogs.

Like it or not, statistics are an essential part of public discourse. Those without proficiency are at a disadvantage. Given the widespread lack of understanding, statistics (invented or otherwise) can be easily manipulated by elites. In the official Republican response to President Obama’s 2012 State of the Union address, for example, Indiana Governor Mitch Daniels depicted unemployment as dire, noting that “nearly half of persons under 30 did not go to work today.” In one sense, Daniels’ statement was true (well, close: it was actually 44.4%). However, the statement is deceptive since it includes people aged 16-22, many of whom are in high school or college. For people ages 25-29 in the labor market, the correct number at the time was closer to 9.7%—high, but not half. Most Americans lack the statistical proficiency to identify the error and know how to find the actual rate. This sort of statistical deception is par for the course in public life today.

The role of public intellectuals of the past was to stand as a bulwark against powerful individuals and institutions by informing the public. Today, Nate Silver’s application of sophisticated statistical tools in accessible, readable ways continues in this watchdog tradition. At the same time, the statistical nature of contemporary political analysis like Silver’s has the potential to widen the gap between the haves and have-nots in their ability to fully participate in democratic society.

Further Reading

Nate Silver’s Greatest Hits

• Forthcoming, 2012. The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t.
• 2009. “How Obama Really Won the Election,” Esquire.
• 2012. “Election Forecast: Obama Begins with Tenuous Advantage,” New York Times Blog FiveThirtyEight.
• 2012. “Models Based on ‘Fundamentals’ Have Failed at Predicting Presidential Elections,” New York Times Blog FiveThirtyEight.
• 2010. “The Most Liveable Neighborhoods in New York: A Quantitative Index of the 50 Most Satisfying Places to Live,” New York Magazine.
• 2009. “Oscar Predictions You Can Bet On! Mr. Statistics, Nate Silver, Goes for the Gold,” New York Magazine. 

Recommended Reading

Joel Best. 2001. Damned Lies and Statistics: Untangling Numbers from the Media, Politicians, and Activists. Best has made a career of pointing out the ways people lie with statistics and helping readers develop the statistical literacy to spot such fibs.

Kaiser Fung. 2010. Numbers Rule Your World: The Hidden Influence of Probabilities and Statistics on Everything You Do. Describes the many exciting, disturbing, and often invisible ways the world is run by statistical information.

Michael Lewis. 2003. Moneyball: The Art of Winning an Unfair Game. The most fascinating book ever written about statistics, this tome explains how stats took over the American pastime.

Caleb Crain. 2007. “Twilight of the Books: What Will Life Be Like if People Stop Reading,” The New Yorker. A widely-read reflection on the decline in reading (and its consequences for the way we think).