Tag Archives: methods/use of data

From Appearance to Identity: How Census Data Collection Changed Race in America

Cross-posted at Global Policy TV.

Publicizing the release of the 1940 U.S. Census data, LIFE magazine released photographs of Census enumerators collecting data from household members.  Yep, Census enumerators. For almost 200 years, the U.S. counted people and recorded information about them in person, by sending out a representative of the U.S. government to evaluate them directly.

By 1970, the government was collecting Census data by mail-in survey. The shift to a survey had dramatic effects on at least one Census category: race.

Before the shift, Census enumerators categorized people into racial groups based on their appearance.  They did not ask respondents how they characterized themselves.  Instead, they made a judgment call, drawing on explicit instructions given to the Census takers.

On a mail-in survey, however, the individual self-identified.  They got to tell the government what race they were instead of letting the government decide.  There were at least two striking shifts as a result of this change:

  • First, it resulted in a dramatic increase in the Native American population.  Between 1980 and 2000, the U.S. Native American population magically grew 110%.  People who had identified as American Indian had apparently been somewhat invisible to the government.
  • Second, to the chagrin of the Census Bureau, 80% of Puerto Ricans choose white (only 40% of them had been identified as white in the previous Census).  The government wanted to categorize Puerto Ricans as predominantly black, but the Puerto Rican population saw things differently.

I like this story.  Switching from enumerators to surveys meant literally shifting our definition of what race is from a matter of appearance to a matter of identity.  And it wasn’t a strategic or philosophical decision. Instead, the very demographics of the population underwent a fundamental unsettling because of the logistical difficulties in collecting information from a large number of people.  Nevertheless, this change would have a profound impact on who we think Americans are, what research about race finds, and how we think about race today.

See also the U.S. Census and the Social Construction of Race and Race and Censuses from Around the World. To look at the questionnaires and their instructions for any decade, visit the Minnesota Population Center.  Thanks to Philip Cohen for sending the link.

—————————

Lisa Wade is a professor of sociology at Occidental College. You can follow her on Twitter and Facebook.

The Stroop Effect: Measuring Our Unconscious

How does a scientist measure your unconscious mind?  It turns out, it can be done.  With a technique called the Implicit Association Test, psychologists can measure your unconscious beliefs about anything: whether, deep down, you associate Black men with weapons, Asians with foreigners, fat people with laziness, men with science, and more.  You can test yourself on all manner of implicit beliefs here.

It works by putting a pair of words on each side of a computer screen. Sometimes the pair matches your unconscious mind; like (for most of us, unfortunately) young and good.  Sometimes the pair challenges your unconscious mind; like (for most of us, unfortunately) old and good.  You’re asked to do a timed test focusing on just one of the pair; we’re all quicker when the terms match than when they don’t.  For more, read up about it here.

In any case, it turns out the phenomenon has a name — the Stroop effect — and the best illustration of it I’ve ever seen was featured on BoingBoing.  It involves colors and color names. For a lifetime, we’ve been taught to associate certain colors with certain names. Accordingly, our brain fires faster and more confidently when we see the name in the color, compared to when we see the name in an opposing color.  See for yourself: can you read both lists of colors equally comfortably, un-self-consciously, and quickly?Probably not.  So, for better or worse, scientists see this same effect when they try to get our brains to process paired words like Asian/American and men/science.  The results of these experiments are depressing (both abstractly and often personally when we take the tests ourselves), but it’s pretty amazing that we’re able to delve that deeply into the mind with such a simple task.

—————————

Lisa Wade is a professor of sociology at Occidental College. You can follow her on Twitter and Facebook.

Critically Examining Scientific Findings in the News

Dmitriy T.M. sent in a TED talk in which Ben Goldacre discusses the problems with many of the scientific findings we hear about in the media, highlighting the importance of scientific literacy and critical consumption of science reporting:

And while we’re on the topic of potentially misleading statistics, Dolores R. and Sarah E. sent in an image posted at boing boing as one of “the best set of infographics ever,” helpfully illustrating the difference between correlation and causation:

Taking Pictures or Making Pictures

Cross-posted at Montclair SocioBlog.

Ethnographers worry that their mere presence on the scene may be influencing what people do and thus compromising the truth of their studies.  They try to minimize that impact, and most of their reports give detailed descriptions of their methods so that readers can assess whether the data might be corrupted.

Photojournalists also claim to be showing us the truth — “pictures don’t lie” — but they compunctions about influencing the people in their photos.  Here for example is a photo taken in Israel by Italian photographer Ruben Salvadori.  (This is a screen grab of a video, hence the subtitles.)

The defiant Palestinian youth, the flames of the roadblock — it’s all very dramatic.  But it is far from spontaneous.  Here’s the photo from another point of view:

Salvadori studied anthropology, and he is well aware that observers influence what they observe.  But editors want “good” photos, not good ethnography.  So observer influence is an asset, not a problem.

If you point a tiny camera at somebody, what is he going to do?  Most likely, he’s going to smile or do something.  Now imagine this enlarged with a group of photographers. That show up with helmets, gas masks, and at least two large cameras each, and they come there to take photos of what you do.  So you’re not going to sit there twiddling your thumbs.

No, the youths don’t twiddle their thumbs, not with the photogs on the scene.  Instead, they burn a flag.

Their relationship is symbiotic.  The photogs want dramatic images, the insurgent youths want publicity.  Of course, even with the Palestinians youths and the Israeli soldiers, when the action gets real, nobody is thinking about how they’ll look in a photo.

—————————

(The full 8-minute video of Salvadori talking about photography in the combat zone was posted at PetaPixel back in October, though I didn’t hear about it until recently.)

Study Shows Home Births Are Not as Safe. So?

Cross-posted at Family Inequality.

There’s an interesting example of how to interpret scientific results — and draw policy implications from them — from the world of birth practices and safety.

The subject of the debate is a major new study from the British Medical Journal. The study followed more than 60,000 women in England with uncomplicated pregnancies, excluding those who had planned caesarean sections and caesarean sections before the start of labor. They compared the number of bad outcomes — from death to broken clavicles – for women depending on where they had their births.

One comparison stands out in the results. From the abstract: “For nulliparous women [those having their first birth], the odds of the primary outcome [that is, any of the negative events] were higher for planned home births” than among those planned for delivery in obstetric units. That is, the home births had higher rates of negative events. The difference is large. Here’s a figure to illustrate:

The error bars show 95% confidence intervals, so you can see the difference between home births and obstetric-unit births is statistically significant at that level. These are the raw comparisons, but the home-versus-obstetric comparison was unchanged when the analysts controlled for age, ethnicity, understanding of English, marital or partner status, body mass index, “deprivation score,” previous pregnancies, and weeks of gestation. Further, by restricting the comparison to uncomplicated pregnancies and excluded all but last-minute c-sections, it seems to be a very strong result.

But what to make of it?

In their conclusion, the authors write:

Our results support a policy of offering healthy nulliparous and multiparous women with low risk pregnancies a choice of birth setting. Adverse perinatal outcomes are uncommon in all settings, while interventions during labour and birth are much less common for births planned in non-obstetric unit settings. For nulliparous women, there is some evidence that planning birth at home is associated with a higher risk of an adverse perinatal outcome.

But in what way do the results “support a policy”? The “higher risks” they found for planned home births are still “uncommon,” by comparison, with those in poor countries, for example. But the home birth risk is 2.7-times greater.

The Skeptical OB, who is a reliable proponent of modern medical births, titled her post, “It’s official: homebirth increases the risk of death.” She added some tables from the supplemental material, showing the type of negative events and conditions that occurred. Her conclusion:

“In other words, any way you choose to look at it, no matter how carefully you slice and dice the data, there is simply no getting around the fact that homebirth increases the risk of perinatal death and brain damage.”

I guess the policy options might include include whether home births should be encouraged, more regulated, covered by public and/or private health insurance, banned, penalized or (further) stigmatized.

Home birth seems safer than letting children ride around unrestrained in the back of pickup trucks, which is legal in North Carolina – as long as they’re engaged in agricultural labor. On the other hand, we have helmet laws for kids on bicycles in many places. And if a child is injured in either situation, hopefully an ambulance would take them to the hospital even if the accident were preventable.

In other words, I don’t think policy questions can be resolved by a comparison of risks, however rigorous.

Now You Will Never Forget What a Venn Diagram Is

Thanks to Flowing Data, we can now enjoy this:

Prescriptions for Poverty: Medical Insecurity as a Measure of Economic Insecurity

Cross-posted at Family Inequality.

Poverty is usually described as a status, as there are people below and above the poverty line. We need to do more to capture and represent the experience of poverty.

There are ways this can be done even in a single survey question, such as this one: ”During the past 12 months, was there any time when you needed prescription medicine but didn’t get it because you couldn’t afford it?” Below are the percentages answering affirmatively, by official poverty-line status.

Percentage of Adults Aged 18-64 Who Did Not Get Needed Prescription Drugs Because of Cost, by Poverty Status (National Health Interview Survey, 1999-2010)

This is not the same as not having any of the prescription drugs you need. What it indicates is economic insecurity rather than deprivation per se, a more nuanced measure than simply being above or below (some percentage of) the poverty line.

Church Saves Marriage… and Produces Curious Coefficients

Cross-posted at Family Inequality.

Things that make you say… “peer review”?

This is the time of year when I expect to read inflated or distorted claims about the benefits of marriage and religion from the National Marriage Project. So I was happy to see the new State of Our Unions report put out by W. Bradford Wilcox’s outfit. My first reading led to a few questions.

First: When they do the “Survey of Marital Generosity” — the privately funded, self-described nationally-representative sample of 18-46-year old Americans, which is the source of this and several other reports, none of them published in any peer-reviewed source I can find — do they introduce themselves to the respondents by saying, “Hello, I’m calling from the Survey of Marital Generosity, and I’d like to ask you a few questions about…” If this were the kind of thing subject to peer review, and I were a reviewer, I would wonder if the respondents were told the name of the survey.

Second: When you see oddly repetitive numbers in a figure showing regression results, don’t you just wonder what’s going on?

Here’s what jumped out at me:

If a student came to my office with these results and said, “Wow, look at the big effect of joint religious practice on marital success,” I’d say, “Those numbers are probably wrong.” I can’t swear they didn’t get exactly the same values for everyone except those couples who both attend religious services regularly — 50 50 50, 13 13 13 , 50 50 50, 21 21 21 — in a regression that adjusts for age, education, income, and race/ethnicity, but that’s only because I don’t swear.*

Of course, the results are beside the point in this report, since the conclusions are so far from the data anyway. From this figure, for example, they conclude:

In all likelihood,  the experience of sharing regular religious attendance — that  is, of enjoying shared rituals that endow one’s marriage with transcendent significance and the support of a community  of family and friends who take one’s marriage seriously— is a solidifying force for marriage in a world in which family life is  increasingly fragile.

OK.

Anyway, whatever presumed error led to that figure seems to reoccur in the next one, at least for happiness:

Just to be clear with the grad student example, I wouldn’t assume the grad student was deliberately cooking the data to get a favorable result, because I like to assume the best about people. Also, people who cook data tend to produce a little more random-looking variation. Also, I would expect the student not to just publish the result online before anyone with a little more expertise had a look at it.

Evidence of a pattern of error is also found in this figure, which also shows predicted percentages that are “very happy,” when age, education, income and race/ethnicity are controlled.

Their point here is that people with lots of kids are happy (which they reasonably suggest may result from a selection effect). But my concern is that the predicted percentages are all between 13% and 26%, while the figures above show percentages that are all between 50% and 76%.

So, in addition to the previous figures probably being wrong, I don’t think this one can be right unless they are wrong. (And I would include “mislabeled” under the heading “wrong,” since the thing is already published and trumpeted to the credulous media.)

Publishing apparently-shoddy work like this without peer review is worse when it happens to support your obvious political agenda. One is tempted to believe that if the error-prone research assistant had produced figures that didn’t conform to the script, someone higher up might have sent the tables back for some error checking. I don’t want to believe that, though, because I like to assume the best about people.

* Just kidding. I do swear.