methods/use of data

“How could we get evidence for this?” I often ask students. And the answer, almost always is, “Do a survey.” The word survey has magical power; anything designated by that name wears a cloak of infallibility.

“Survey just means asking a bunch of people a bunch of questions,” I’ll say. “Whether it has any value depends on how good the bunch of people is and how good the questions are.”  My hope is that a few examples of bad sampling and bad questions will demystify.

For example, Variety:

11

Here’s the lede:

Despite its Biblical inspiration, Paramount’s upcoming “Noah” may face some rough seas with religious audiences, according to a new survey by Faith Driven Consumers.

The data to confirm that idea:

The religious organization found in a survey that 98% of its supporters were not “satisfied” with Hollywood’s take on religious stories such as “Noah,” which focuses on Biblical figure Noah.

The sample:

Faith Driven Consumers surveyed its supporters over several days and based the results on a collected 5,000+ responses.

And (I’m saving the best till last) here’s the crucial survey question:

As a Faith Driven Consumer, are you satisfied with a Biblically themed movie — designed to appeal to you — which replaces the Bible’s core message with one created by Hollywood?

As if the part about “replacing the Bible’s core message” weren’t enough, the item reminds the respondent of her or his identity as a Faith Driven Consumer. It does make you wonder about that 2% who either were fine with the Hollywood* message or didn’t know.

You can’t really fault Faith Driven Consumer too much for this shoddy “research.” They’re not in business to find the sociological facts. What’s appalling is that Variety accepts it at face value and without comment.

Cross-posted at Montclair SocioBlog.

Jay Livingston is the chair of the Sociology Department at Montclair State University. You can follow him at Montclair SocioBlog or on Twitter.

Let’s imagine that a woman — we’ll call her “you,” like they do in relationship advice land — is trying to calculate the odds that a man will call back after sex. Everyone tells you that if you sleep with a guy on the first date he is less likely to call back. The theory is that giving sex away at a such a low “price” lowers the man’s opinion of you, because everyone thinks sluts are disgusting.* Also, shame on you.

So, you ask, does the chance he will call back improve if you wait till more dates before having sex with him? You ask around and find that this is actually true: The times you or your friends waited till the seventh date, two-thirds of the guys called back, but when you slept with him on the first date, only one-in-five called back. From the data, it sure looks like sleeping with a guy on the first date reduces the odds he’ll call back.

1 (2)

So, does this mean that women make men disrespect them by having sex right away? If that’s true, then the historical trend toward sex earlier in relationships could be really bad for women, and maybe feminism really is ruining society.

Like all theories, this one assumes a lot. It assumes you (women) decide when couples will have sex, because it assumes men always want to, and it assumes men’s opinion of you is based on your sexual behavior. With these assumptions in place, the data appear to confirm the theory.

But what if that those assumptions aren’t true? What if couples just have more dates when they enjoy each other’s company, and men actually just call back when they like you? If this is the case, then what really determines whether the guy calls back is how well-matched the couple is, and how the relationship is going, which also determines how many dates you have.

What was missing in the study design was relationship survival odds. Here is a closer look at the same data (not real data), with couple survival added:

5
(Graph corrected from an earlier version.)

.

By this interpretation, the decision about when to have sex is arbitrary and doesn’t affect anything. All that matters is how much the couple like and are attracted to each other, which determines how many dates they have, and whether the guy calls back. Every couple has a first date, but only a few make it to the seventh date. It appears that the first-date-sex couples usually don’t last because people don’t know each other very well on first dates and they have a high rate of failure regardless of sex. The seventh-date-sex couples, on the other hand, usually like each other more and they’re very likely to have more dates. And: there are many more first-date couples than seventh-date couples.

So the original study design was wrong. It should have compared call-back rates after first dates, not after first sex. But when you assume sex runs everything, you don’t design the study that way. And by “design the study” I mean “decide how to judge people.”

I have no idea why men call women back after dates. It is possible that when you have sex affects the curves in the figure, of course. (And I know even talking about relationships this way isn’t helping.) But even if sex doesn’t affect the curves, I would expect higher callback rates after more dates.

Anyway, if you want to go on blaming everything bad on women’s sexual behavior, you have a lot of company. I just thought I’d mention the possibility of a more benign explanation for the observed pattern that men are less likely to call back after sex if the sex takes place on the first date.

* This is not my theory.

Cross-posted at Family Inequality and Pacific Standard.

Philip N. Cohen is a professor of sociology at the University of Maryland, College Park, and writes the blog Family Inequality. You can follow him on Twitter or Facebook.

A survey question is only as good as its choices. Sometimes an important choice has been left off the menu.  I was Gallup polled once, long ago. I’ve always felt that they didn’t get my real opinion.

“What’d they ask?” said my brother when I mentioned it to him.

“You know, they asked whether I approved of the way the President was doing his job.”  Nixon – this was in 1969.
“What’d you say?”

“I said I disapproved of his entire existential being.”

I was exaggerating my opinion, and I didn’t actually say that to the pollster.  But even if I had, my opinion would have been coded as “disapprove.”

For many years the American National Election Study has asked:

How much of the time do you think you can trust the government in Washington to do what is right – just about always, most of the time or only some of the time?

The trouble with these choices at that they exclude the truly disaffected. The worst you can say about the federal government is that it can be trusted “only some of the time.”  A few ornery souls say they don’t trust the federal at all. But because that view is a write-in candidate, it usually gets only one or two percent of the vote.

This year the study included “never” in the options read to respondents.  Putting “no-way, no-how” right there on the ballot makes a big difference. And as you’d expect, there were party differences:

1 (2)

Over half of Republicans say that the federal government can NEVER be trusted.

The graph appears in this Monkey Cage post by Marc Hetherington and Thomas Rudolph. Of course, some of those “never” Republicans don’t really mean “never ever.”  If a Republican becomes president, they’ll become more trusting, and the “never-trust” Democrat tide will rise.  Here’s the Hetherington-Rudolph graph tracking changes in the percent of people who do trust Washington during different administrations.

1 (3)

This one seems to show three things:

  1. Trust took a dive in the 1960s and 70s and never really recovered.
  2. Republican trust is much more volatile, with greater fluctuations depending on which party is in the White House.
  3. Republicans really, really hate President Obama.

Cross-posted at Montclair SocioBlog.

Jay Livingston is the chair of the Sociology Department at Montclair State University. You can follow him at Montclair SocioBlog or on Twitter.

Last week Kay Hymowitz (who sometimes works out of a PO Box rented by Brad Wilcox) wrote the following in the LA Times:

As far back as the 1970s, family researchers began noticing that… [b]oys from broken homes were more likely than their peers to get suspended and arrested… And justice experts have long known that juvenile facilities and adult jails overflow with sons from broken families. Liberals often assume that these kinds of social problems result from our stingy support system for single mothers and their children. But the link between criminality and fatherlessness holds even in countries with lavish social welfare systems.

Ah, the link between criminality and fatherlessness again. So ingrained is the assumption that crime rates always go up that conservatives making this argument do not even see the need to account for the incredible, world-historical drop in violence that has accompanied the collapse of the nuclear family. I know Kay Hymowitz knows this, because we’ve argued about it before. But if her editors and readers don’t, why should she make a big deal out of it?

In this graph I show the scales down to zero so you can see the proportional change in each trend: father-not-present boys ages 10-14 and male juvenile violent-crime arrest rates.

1

I’m not arguing about whether boys living without fathers are more likely to commit crimes. I’m just saying that this is very unlikely to be the major cause of male juvenile violent crime if the trends can move so drastically in opposite directions at the same time. These aren’t little fluctuations. Even if you leave out the late-80s-early-90s spike in crime, arrests fell about 40% from 1980 to 2010 while father-absent boys increased almost 50%.

If you are going to argue for a strong association — which Hymowitz does — and use words like “tide,” you should at least acknowledge that the problem you are trumpeting is getting better while the cause you are bemoaning is getting worse.

Cross-posted at Family Inequality.

Philip N. Cohen is a professor of sociology at the University of Maryland, College Park, and writes the blog Family Inequality. You can follow him on Twitter or Facebook.

The Federal Reserve Bank has said it will maintain its stimulus policy as long as the economy remains weak. One of its key indicators for the strength of the economy is the unemployment rate, which has been steadily falling for several years, from 10% in October 2009 to 7.3% in August 2013.  However, this decline in the official unemployment rate gives a misleading picture of economic conditions, at least as far as the labor market is concerned.

The reason, as the Economy Policy Institute explains, is because of the large number of “missing workers.”  These missing workers are…

…potential workers who, because of weak job opportunities, are neither employed nor actively seeking a job. In other words, these are people who would be either working or looking for work if job opportunities were significantly stronger. Because jobless workers are only counted as unemployed if they are actively seeking work, these “missing workers” are not reflected in the unemployment rate.

We are seeing many more missing workers now than in recent history.  The chart below shows the Economic Policy Institute estimate for the number of missing workers.

missing workers_Page_1

 The next chart compares the estimated unemployment rate including missing workers (in orange) with the official unemployment rate (in blue).

unemployment rates_Page_1

As you can see, while the official unemployment rate continues to decline, the corrected unemployment rate remains stuck at a rate above 10%. In other words labor market conditions remain dismal. And here we are only talking about employment.  If we consider the quality of the jobs being created, things are even worse.

Martin Hart-Landsberg is a professor of economics at Lewis and Clark College. You can follow him at Reports from the Economic Front.

In survey questions, the result you get might depend on the choices you offer.

An article at The Atlantic explains “Why Americans All Believe They’re Middle Class.”  But is that what we all believe?  The author, Anat Shenker-Osorio, started with from these figure from a September 2012 Pew report.
1
Only 8-9% of Americans put themselves in the lower or upper class.  The other 91% say that they are “middle class,” some with a modifier (upper or lower), some without.  Shenker-Osorio continues:

Researching how people’s unconscious assumptions affect their perception of economic issues, I explored the linguistic dynamics behind the term “middle class,” especially in comparison to other economic groupings.

That would be fine, except that both she and Pew made one huge omission.  The Pew survey didn’t include “working class” as an option.  Out of sight, out of unconscious assumptions.

Language and Surveys

How big an omission is this?  Since 1972, the GSS has asked a similar question to tap “subjective social class” (i.e., what class people think they are regardless of their objective circumstances).  But the GSS includes “working” along with the upper, middle, and lower.

2

Like the Pew survey, the GSS finds less than 10% putting themselves in the upper or lower class.  But for the past forty years, the remaining nine-tenths of the population have been evenly split between “working” and “middle.”

Shenker-Osorio’s linguistic analysis runs into other data conflicts.  It’s not always easy to know what Americans mean by upper, lower, or middle class because:

Americans are relatively skittish about mentioning class. Contrasting databases of text from U.S. and UK sources, we find that Brits use “upper class” and “lower class” more readily; we prefer “wealthy” and “poor.”

But another database, the books in Google nGrams, shows something much different.

Contrasting Data

I constructed a ratio of American to British for the terms “upper class” and “lower class.”  A ratio of more than 100% means that the term appeared more frequently in American books.

Ratio for “upper class”:

3

Ratio for “lower class”:

4

In general, since 1900, US and UK books used these terms at about the same frequency.  But from 1955-1965, the US heard a crescendo in class talk.  By 1965, US books mentioned the “lower class” four times as often as did UK books.  Since then class talk in the US declined as rapidly as it had increased. (For some reason, Shenker-Osorio was unaware of my earlier post on these matters.)

The real US-UK difference is in “working class,” a term that Shenker-Osorio ignores. Since 1935, it has appeared less frequently in US books.  For the last 30 years, British books have mentioned the working class twice as often.

Ratio for “working class”:

5

It may be that the databases Shenker-Osorio used are better than nGrams, and it’s frustrating to find different sources of data pointing in different directions.  More important, we still don’t know what people mean when they say they are middle class.  Shenker-Osorio sees it as a category of exclusion.  The images we have of upper and lower are so extreme as to apply to almost nobody.

Not finding popular depictions of wealth and poverty similar to our own lived experiences, we determine we must be whatever’s left over.

True perhaps, but it tells only what people think middle class is not. I’m not familiar with the research on subjective social class, but it seems that we still don’t know what people think “middle class” actually is.  Nor do we know what they have in mind when they say they are working class.  I have my own hunches, but I will leave them for a later post.

Cross-posted at Montclair SocioBlog.

Jay Livingston is the chair of the Sociology Department at Montclair State University. You can follow him at Montclair SocioBlog or on Twitter.

Cross-posted at Reports from the Economic Front.

Wealth data is not easy to get.  Still for three years now, Credit Suisse Research Institute has published an annual Global Wealth Databook which attempts to estimate global wealth holdings.  The most recent issue includes data covering 2012.  According to Credit Suisse, the goal “is to provide the best available estimates of the wealth holdings of households around the world for the period since the year 2000.”

According to the publication, global household wealth was $222.7 trillion in mid-2012, equal to $48,500 for each of the 4.6 billion adults in the world.  Wealth is defined as “the marketable value of financial assets plus non-financial assets (principally housing and land) less debts.”

Not surprisingly, as the figure below shows, average global wealth varies considerably across countries and regions.

picture wealth

Also significant are the values of the mean vs the median wealth in each of the countries.  Mean or average wealth is calculated by dividing the total wealth of a country by its adult population.   Median wealth is the wealth holdings of the adult in the middle of the wealth distribution. The median is generally considered a far more reliable indicator of wealth because it is less sensitive to extremes at the top or bottom of the distribution.  The greater the divergence of mean and median wealth, the greater is the wealth inequality.

The table below provides mean and median wealth estimates for those countries with generally reliable data. As you can see, the U.S. ranks high in terms of mean wealth, trailing only 5 countries.  Things are quite different when it comes to median wealth; the U.S. trails 26 countries!  Not surprisingly, then, the U.S. is No. 1 when it comes to the mean/median wealth ratio, or wealth inequality.

global wealth

We clearly dominate in the number of millionaires and the upper global wealth categories.  Are we a wealthy country? Definitely.  Is that wealth concentrated in relatively few hands?  Definitely.

Martin Hart-Landsberg is a professor of economics at Lewis and Clark College. You can follow him at Reports from the Economic Front.

Cross-posted at Montclair SocioBlog.

Forty years ago Richard Easterlin proposed the paradox that people in wealthier countries were no happier than those in less wealthy countries.  Subsequent research on money and happiness brought modifications and variations, notably that within a single country, while for the poor, more money meant fewer problems, for the wealthier people — those with enough or a bit more — enough is enough.  Increasing your income from $100,000 to $200,000 isn’t going to make you happier.

It was nice to hear researchers singing the same lyrics we’ll soon be hearing in commencement speeches and that you hear in Sunday sermons and pop songs (“the best things in life are free”; “mo’ money mo’ problems”).  But this moral has a sour-grapes taste; it’s a comforting fable we non-wealthy tell ourselves all the while suspecting that it probably isn’t true.

A recent Brookings paper by Betsey Stevenson and Justin Wolfers adds to that suspicion.  Looking at comparisons among countries and within countries, they find that when it comes to happiness, you can never be too rich.

1

Stevenson and Wolfers also find no “satiation point,” some amount where happiness levels off despite increases in income.  They provide US data from a 2007 Gallup survey:

2

The data are pretty convincing.  Even as you go from rich to very rich, the proportion of “very satisfied” keeps increasing.  (Sample size in the stratosphere might be a problem: only 8 individuals reported annual incomes over $500,000;100% of them, though, were “very happy.”)

Did Biggie and Alexis get it wrong?

Around the time that the Stevenson-Wolfers study was getting attention in the world beyond Brookings, I was having lunch with a friend who sometimes chats with higher ups at places like hedge funds and Goldman Sachs.  He hears wheeler dealers complaining about their bonuses. “I only got ten bucks.”  Stevenson and Wolfers would predict that this guy’s happiness would be off the charts given the extra $10 million.  But he does not sound like a happy master of the universe.

I think that the difference is more than just the clash of anecdotal and systematic evidence.  It’s about defining and measuring happiness.  The Stevenson-Wolfers paper uses measures of “life satisfaction.”  Some surveys ask people to place themselves on a ladder according to “how you feel about your life.”  Others ask

All things considered, how satisfied are you with your life as a whole these days?

The GSS uses happy instead of satisfied, but the effect is the same:

Taken all together, how would you say things are these days – would you say that you are very happy, pretty happy, or not too happy?

When people hear these questions, they may think about their lives in a broader context and compare themselves to a wider segment of humanity.  I imagine that Goldman trader griping about his “ten bucks” was probably thinking of the guy down the hall who got twelve.  But when the survey researcher asks him where he is on that ladder, he may take a more global view and recognize that he has little cause for complaint.  Yet moment to moment during the day, he may look anything but happy.  There’s a difference between “affect” (the preponderance of momentary emotions) and overall life satisfaction.

Measuring affect is much more difficult — one method requires that people log in several times a day to report how they’re feeling at that moment — but the correlation with income is weaker.

In any case, it’s nice to know that the rich are benefitting from getting richer.  We can stop worrying about their being sad even in their wealthy pleasure and turn our attention elsewhere.  We got 99 problems, but the rich ain’t one.

Jay Livingston is the chair of the Sociology Department at Montclair State University. You can follow him at Montclair SocioBlog or on Twitter.