methods/use of data

Want to help fight fake news and manage political panics? We have to learn to talk about numbers.

While teaching basic statistics to sociology undergraduates, one of the biggest trends I noticed was students who thought they hated math experiencing a brain shutdown when it was time to interpret their results. I felt the same way when I started in this field, and so I am a big advocate for working hard to bridge the gap between numeracy and literacy. You don’t have to be a statistical wizard to make your reporting clear to readers.

Sociology is a great field to do this, because we are used to going out into the world and finding all kinds of cultural tropes (like pointlessly gendered products!). My new favorite trope is the Half-Dozen Headline. You can spot them in the wild, or through Google News with a search for “half dozen.” Every time I read one of these headlines, my brain echoes with “half of a dozen is six.”

Sometimes, six is a lot:

Sometimes, six is not:

(at least, not relative to past administrations)

Sometimes, well, we just don’t know:

Is this five deaths (nearly six)? Is a rate of about two deaths a year in a Walmart parking lot high? If people already struggle to interpret raw numbers, wrapping your findings in fuzzy language only makes the problem worse.

Spotting Half-Dozen Headlines is a great introductory exercise for classes in social statistics, public policy, journalism, or other fields that use applied data analysis. If you find a favorite Half-Dozen Headline, be sure to send it our way!

Evan Stewart is a Ph.D. candidate in sociology at the University of Minnesota. You can follow him on Twitter.

The Star Tribune recently ran an article about a new study from George Washington University tracking cases of Americans who traveled to join jihadist groups in Syria and Iraq since 2011. The print version of the article was accompanied by a graph showing that Minnesota has the highest rate of cases in the study. TSP editor Chris Uggen tweeted the graph, noting that this rate represented a whopping seven cases in the last six years.

Here is the original data from the study next to the graph that the paper published:

(Click to Enlarge)

Social scientists often focus on rates when reporting events, because it make cases easier to compare. If one county has 300 cases of the flu, and another has 30,000, you wouldn’t panic about an epidemic in the second county if it had a city with many more people. But relying on rates to describe extremely rare cases can be misleading. 

For example, the data show this graph misses some key information. California and Texas had more individual cases than Minnesota, but their large populations hide this difference in the rates. Sorting by rates here makes Minnesota look a lot worse than other states, while the number of cases is not dramatically different. 

As far as I can tell, this chart only appeared in the print newspaper photographed above and not on the online story. If so, this chart only went to print audiences. Today we hear a lot of concern about the impact of “filter bubbles,” especially online, and the spread of misleading information. What concerns me most about this graph is how it shows the potential impact of offline filter bubbles in local communities, too.

Evan Stewart is a Ph.D. candidate in sociology at the University of Minnesota. You can follow him on Twitter.

That large (and largely trademarked) sporting event is this weekend. In honor of its reputation for massive advertising, Lisa Wade tipped me off about this interesting content analysis of last year’s event by the Media Education Foundation.

MEF watched last year’s big game and tallied just how much time was devoted to playing and how much was devoted to ads and other branded content during the game. According to their data, the ball was only in play “for a mere 18 minutes and 43 seconds, or roughly 8% of the entire broadcast.”

MEF used a pie chart to illustrate their findings, but readers can get better information from comparing different heights instead of different angles. Using their data, I quickly made this chart to more easily compare branded and non-branded content.

Data Source: Media Education Foundation, 2018

One surprising thing that jumps out of this data is that, for all the hubbub about commercials, far and away the most time is devoted to replays, shots of the crowd, and shots of the field without the ball in play. We know “the big game” is a big sell, but it is interesting to see how the thing it sells the most is the spectacle of the event itself.

Evan Stewart is a Ph.D. candidate in sociology at the University of Minnesota. You can follow him on Twitter.

Based on analyses of General Social Survey data, a well-designed and respected source of data about American life, members of the Millennial generation are acquiring about the same number of sexual partners as the Baby Boomers. This data suggests that the big generational leap was between the Boomers and the generation before them, not the Boomers and everyone that came after. And rising behavioral permissiveness definitely didn’t start with the Millennials. Sexually speaking, Millennials look a lot like their parents at the same age and are perhaps even less sexually active then Generation X.

Is it true?

It doesn’t seem like it should be true. In terms of attitudes, American society is much more sexually permissive than it was for Boomers, and Millennials are especially more permissive. Boomers had to personally take America through the sexual revolution at a time when sexual permissiveness was still radical, while Generation X had to contend with a previously unknown fatal sexually transmitted pandemic. In comparison, the Millennials have it so easy. Why aren’t they having sex with more people?

A new study using data from the National Survey of Family Growth (NSFG) (hat tip Paula England) contrasts with previous studies and reports an increase. It finds that nine out of ten Millennial women had non-marital sex by the time they were 25 years old, compared to eight out of ten Baby Boomers. And, among those, Millennials reported two additional total sexual partners (6.5 vs. 4.6).

Nonmarital Sex by Age 25, Paul Hemez

Are Millennials acquiring more sexual partners after all?

I’m not sure. The NSFG report used “early” Millennials (only ones born between 1981 and 1990). In a not-yet-released book, the psychologist Jean Twenge uses another survey — the Youth Risk Behavior Surveillance System — to argue that the next generation (born between 1995 and 2002), which she calls the “iGen,” are even less likely to be sexually active than Millennial. According to her analysis, 37% of 9th graders in 1995 (born in 1981, arguably the first Millennial year) had lost their virginity, compared to 34% in 2005, and 24% in 2015.

Percentage of high school students who have ever had sex, by grade. Youth Risk Behavior Surveillance System, 1991-2015.

iGen, Jean Twenge

If Twenge is right, then we’re seeing a decline in the rate of sexual initiation and possibly partner acquisition that starts somewhere near the transition between Gen X and Millennial, proceeds apace throughout the Millennial years, and is continuing — Twenge argues accelerating — among the iGens. So, if the new NSFG report finds an increase in sexual partners between the Millennials and the Boomers, it might be because they sampled on “early” Millennials, those closer to Gen Xers, on the top side of the decline.

Honestly, I don’t know. It’s interesting though. And it’s curious why the big changes in sexually permissive attitudes haven’t translated into equally sexually permissive behaviors. Or, have actually accompanied a decrease in sexual behavior. It depends a lot on how you chop up the data, too. Generations, after all, all artificial categories. And variables like “nonmarital sex by age 25” are specific and may get us different findings than other measures. Sociological questions have lots of moving parts and it looks as if we’re still figuring this one out.

Lisa Wade, PhD is a professor at Occidental College. She is the author of American Hookup, a book about college sexual culture, and a textbook about gender. You can follow her on Twitter, Facebook, and Instagram.

Originally posted at Scatterplot.

There are few things more satisfying than finding another reason that millennials are the worst. They’re narcissistic, coddled, unpatriotic, racist, and nervous about free speech. And now, millennial men want a return to the nostalgic 1950s, with women in the kitchen, whipping up a nice quiche after a hard day on the line.

This is the story presented in Stephanie Coontz’s Friday piece in the New York Times, “Do Millennial Men Want Stay-at-Home Wives,”which reports on evidence from the Council on Contemporary Families (using the General Social Survey) and from sociologists Joanna Pepin and David Cotter (using Monitoring the Future ).

Journalists have gone a bit nuts for this millennial-as-Ward-Cleaver narrative, consistent with what we already know about garbage millennials, and stories from Quartz and Time Magazine have already popped up.

The Times piece includes this damning trend among men ages 18-25:

See? Millennial men are the WORST.


But the GSS just released their 2016 data this week. 89% of men disagree or strongly disagree with the statement “It is much better for everyone involved if the man is the achiever outside the home and the women takes care of the home and family” – the highest rate among either men or women ages 18-25 in the GSS’s 40-year history. It’s also much higher than the rate reported by everyone older than 25, about 71%.

So is the story, “Clinton defeat inspires millennial men to gender equality”? Or more likely, “Garbage millennial men can’t make up their mind about women”?

I suspect it’s another, less sexy story: you can’t say a lot about millennials based on talking to 66 men.

The GSS surveys are pretty small – about 2,000-3,000 per wave – so once you split by sample, and then split by age, and then exclude the older millennials (age 26-34) who don’t show any negative trend in gender equality, you’re left with cells of about 60-100 men ages 18-25 per wave. Standard errors on any given year are 6-8 percent.

So let’s throw some statistics at it. Suppose you want to know whether there is a downward trend in young male disagreement with the women-in-the-kitchen statement. Using all available GSS data, there is a positive, not statistically significant trend in men’s attitudes (more disagreement). Starting in 1988 only, there is very, very small negative, not statistically significant effect.

Only if we pick 1994 as a starting point, as Coontz does, ignoring the dip just a few years prior, do we see a negative less-than half-percentage point drop in disagreement per year, significant at the 10-percent level.

As Columbia statistician Andrew Gelman wisely warns, none of these results account for the many, many paths the researchers could have taken to arrive at these results, which can make overreliance on any of these p-values problematic. For example, if we just looked at millennials the way they’re usually defined, as individuals ages 18-34?

The Pepin and Cotter piece, in fact, presents two additional figures in direct contrast with the garbage millennial theory – in Monitoring the Future, millennial men’s support for women in the public sphere has plateaued, not fallen; and attitudes about women working have continued to improve, not worsen. Their conclusion is, therefore, that they find some evidence of a move away from gender equality – a nuance that’s since been lost in the discussion of their work.

So what does this mean? Standard errors matter, and millennials might not always be as garbage as we think they are.

Emily Beam is Assistant Professor of Economics at the University of Vermont. She studies labor and development economics, with a particular focus on employment and education policy, migration, fertility and marriage, and the role of incomplete information and behavioral biases on individual decision-making.

Flashback Friday.

Add to the list of new books to read Delusions of Gender: How Our Minds, Society, and Neurosexism Create Difference, by Cordelia Fine. Feeding my interest in the issue of sexual dimorphism in humans — which we work so hard to teach to children — the book is described like this:

Drawing on the latest research in neuroscience and psychology, Cordelia Fine debunks the myth of hardwired differences between men’s and women’s brains, unraveling the evidence behind such claims as men’s brains aren’t wired for empathy and women’s brains aren’t made to fix cars.

Good reviews here and here report that Fine tackles an often-cited study of newborn infants’ sex difference in preferences for staring at things, by Jennifer Connellan and colleagues in 2000. They reported:

…we have demonstrated that at 1 day old, human neonates demonstrate sexual dimorphism in both social and mechanical perception. Male infants show a stronger interest in mechanical objects, while female infants show a stronger interest in the face.

And this led to the conclusion: “The results of this research clearly demonstrate that sex differences are in part biological in origin.” They reached this conclusion by alternately placing Connellan herself or a dangling mobile in front of tiny babies, and timing how long they stared. There is a very nice summary of problems with the study here, which seriously undermine its conclusion.

However, even if the methods were good, this is a powerful example of how a tendency toward difference between males and females is turned into a categorical opposition between the sexes — as in, the “real differences between boys and girls.”

To illustrate this, here’s a graphic look at the results in the article, which were reported in this table:

They didn’t report the whole distribution of boys’ and girls’ gaze-times, but it’s obvious that there is a huge overlap in the distributions, despite a difference in the means. In the mobile-gaze-time, for example, the difference in averages is 9.7 seconds, while the standard deviations are more than 20 seconds. If I turn to my handy normal curve spreadsheet template, and fit it with these numbers, you can see what the pattern might look like (I truncate these at 0 seconds and 70 seconds, as they did in the study):

Source: My simulation assuming normal distributions from the data in the table above.

All I’m trying to say is that the sexes aren’t opposites, even if they have some differences that precede socialization.

If you could show me that the 1-day-olds who stare at the mobiles for 52 seconds are more likely to be engineers when they grow up than the ones who stare at them for 41 seconds (regardless of their gender) then I would be impressed. But absent that, if you just want to use such amorphous differences at birth to explain actual segregation among real adults, then I would not be impressed.

Originally posted in September, 2010.

Philip N. Cohen is a professor of sociology at the University of Maryland, College Park. He writes the blog Family Inequality and is the author of The Family: Diversity, Inequality, and Social Change. You can follow him on Twitter or Facebook.

Flashback Friday.

Bewildered by Nazi soldiers’ willingness to perpetuate the horrors of World War II, Stanley Milgram set out to test the extent to which average people would do harm if instructed by an authority figure. In what would end up being one of the most famous studies in the history of social psychology, the experimenter would instruct study subjects to submit a heard, but unseen stranger (who was reputed to have a heart condition) to a series of increasingly strong electric shocks. The unseen stranger (actually a tape recording) would yelp and cry and scream and beg… and eventually be silent. If the study subject expressed a desire to quit administering the shocks, the experimenter would prod four times:

1. Please continue.
2. The experiment requires that you continue.
3. It is absolutely essential that you continue.
4. You have no other choice, you must go on.

If, after four prods, the subject still refused to administer the shock, the experiment was over.

In his initial study, though all participants at some point required prodding, 65 percent of people (26 out of 40) continued to submit the stranger to electric shocks all the way up to (a fake) 450-volts, a dose that was identified as fatal and was administered after the screaming turned to silence. You can watch a BBC replication of the studies.

Originally posted in August 2010.

Lisa Wade, PhD is a professor at Occidental College. She is the author of American Hookup, a book about college sexual culture, and a textbook about gender. You can follow her on Twitter, Facebook, and Instagram.

TW: racism  and sexual violence; originally posted at Family Inequality.

I’ve been putting off writing this post because I wanted to do more justice both to the history of the Black-men-raping-White-women charge and the survey methods questions. Instead I’m just going to lay this here and hope it helps someone who is more engaged than I am at the moment. I’m sorry this post isn’t higher quality.

Obviously, this post includes extremely racist and misogynist content, which I am showing you to explain why it’s bad.

This is about this very racist meme, which is extremely popular among extreme racists.


The modern racist uses statistics, data, and even math. They use citations. And I think it takes actually engaging with this stuff to stop it (this is untested, though, as I have no real evidence that facts help). That means anti-racists need to learn some demography and survey methods, and practice them in public. I was prompted to finally write on this by a David Duke video streamed on Facebook, in which he used exaggerated versions of these numbers, and the good Samaritans arguing with him did not really know how to respond.

For completely inadequate context: For a very long time, Black men raping White women has been White supremacists’ single favorite thing. This was the most common justification for lynching, and for many of the legal executions of Black men throughout the 20th century. From 1930 to 1994 there were 455 people executed for rape in the U.S., and 89% of them were Black (from the 1996 Statistical Abstract):


For some people, this is all they need to know about how bad the problem of Blacks raping Whites is. For better informed people, it’s the basis for a great lesson in how the actions of the justice system are not good measures of the crimes it’s supposed to address.

Good data gone wrong

Which is one reason the government collects the National Crime Victimization Survey (NCVS), a large sample survey of about 90,000 households with 160,000 people. In it they ask about crimes against the people surveyed, and the answers the survey yields are usually pretty different from what’s in the crime report statistics – and even further from the statistics on things like convictions and incarceration. It’s supposed to be a survey of crime as experienced, not as reported or punished.

It’s an important survey that yields a lot of good information. But in this case the Bureau of Justice Statistics is doing a serious disservice in the way they are reporting the results, and they should do something about it. I hope they will consider it.

Like many surveys, the NCVS is weighted to produce estimates that are supposed to reflect the general population. In a nutshell, that means, for example, that they treat each of the 158,000 people (over age 12) covered in 2014 as about 1,700 people. So if one person said, “I was raped,” they would say, “1700 people in the US say they were raped.” This is how sampling works. In fact, they tweak it much more than that, to make the numbers add up according to population distributions of variables like age, sex, race, and region – and non-response, so that if a certain group (say Black women) has a low response rate, their responses get goosed even more. This is reasonable and good, but it requires care in reporting to the general public.

So, how is the Bureau of Justice Statistics’ (BJS) reporting method contributing to the racist meme above? The racists love to cite Table 42 of this report, which last came out for the 2008 survey. This is the source for David Duke’s rant, and the many, many memes about this. The results of Google image search gives you a sense of how many websites are distributing this:


Here is Table 42, with my explanation below:


What this shows is that, based on their sample, BJS extrapolates an estimate of 117,640 White women who say they were sexually assaulted, or threatened with sexual assault, in 2008 (in the red box). Of those, 16.4% described their assailant as Black (the blue highlight). That works out to 19,293 White women sexually assaulted or threatened by Black men in one year – White supremacists do math. In the 2005 version of the table these numbers were 111,490 and 33.6%, for 37,460 White women sexually assaulted or threatened by Black men, or:


Now, go back to the structure of the survey. If each respondent in the survey counts for about 1,700 people, then the survey in 2008 would have found 69 White women who were sexually assaulted or threatened, 11 of whom said their assailant was Black (117,640/1,700). Actually, though, we know it was less than 11, because the asterisk on the table takes you to the footnote below which says it was based on 10 or fewer sample cases. In comparison, the survey may have found 27 Black women who said they were sexually assaulted or threatened (46,580/1,700), none of whom said their attacker was White, which is why the second blue box shows 0.0. However, it actually looks like the weights are bigger for Black women, because the figure for the percentage assaulted or threatened by Black attackers, 74.8%, has the asterisk that indicates 10 or fewer cases. If there were 27 Black women in this category, then 74.8% of them would be 20. So this whole Black women victim sample might be as little as 13, with bigger weights applied (because, say, Black women had a lower response rate). If in fact Black women are just as likely to be attacked or assaulted by White men as the reverse, 16%, you might only expect 2 of those 13 to be White, and so finding a sample 0 is not very surprising. The actual weighting scheme is clearly much more complicated, and I don’t know the unweighted counts, as they are not reported here (and I didn’t analyze the individual-level data).

I can’t believe we’re talking about this. The most important bottom line is that the BJS should not report extrapolations to the whole population from samples this small. These population numbers should not be on this table. At best these numbers are estimated with very large standard errors. (Using a standard confident interval calculator, that 16% of White women, based on a sample of 69, yields a confidence interval of +/- 9%.) It’s irresponsible, and it’s inadvertently (I assume) feeding White supremacist propaganda.

Rape and sexual assault are very disturbingly common, although not as common as they were a few decades ago, by conventional measures. But it’s a big country, and I don’t doubt lots of Black men sexual assault or threaten White women, and that White men sexually assault or threaten Black women a lot, too – certainly more than never. If we knew the true numbers, they would be bad. But we don’t.

A couple more issues to consider. Most sexual assault happens within relationships, and Black women have interracial relationships at very low rates. In round numbers (based on marriages), 2% of White women are with Black men, and 5% of Black women are with White men, which – because of population sizes – means there are more than twice as many couples with Black-man/White-woman than the reverse. At very small sample sizes, this matters a lot. But we would expect there to be more Black-White rape than the reverse based on this pattern alone. Consider further that the NCVS is a householdsample, which means that if any Black women are sexually assaulted by White men in prison, it wouldn’t be included. Based on a 2011-2012 survey of prison and jail inmates, 3,500 women per year are the victim of staff sexual misconduct, and Black women inmates were about 50% more likely to report this than White women. So I’m guessing the true number of Black women sexually assaulted by White men is somewhat greater than zero, and that’s just in prisons and jails.

The BJS seems to have stopped releasing this form of the report, with Table 42, maybe because of this kind of problem, which would be great. In that case they just need to put out a statement clarifying and correcting the old reports – which they should still do, because they are out there. (The more recent reports are skimpier, and don’t get into this much detail [e.g., 2014] – and their custom table tool doesn’t allow you to specify the perceived race of the offender).

So, next time you’re arguing with David Duke, the simplest response to this is that the numbers he’s talking about are based on very small samples, and the asterisk means he shouldn’t use the number. The racists won’t take your advice, but it’s good for everyone else to know.

Philip N. Cohen is a professor of sociology at the University of Maryland, College Park. He writes the blog Family Inequality and is the author of The Family: Diversity, Inequality, and Social Change. You can follow him on Twitter or Facebook.