This week, my manuscript, co-authored by Melissa Lindquist and Julie Arnold, “Million Dollar Maybe? The Effect of Female Presence in Movies on Box Office Returns” was published online by Sociological Inquiry. It will appear in print later this year. Cross-posted at my web site.

In 2012, I wrote an essay for The Society Pages about how Nate Silver of FiveThirtyEight (at the time, hosted by the New York Times and now by ESPN) was carving out a new “blogger sociology.” As I wrote then, Silver writes for a new technocratic audience and produces posts with “outputs from multivariate regression analyses, resplendent with unstandardized coefficients, standard errors, and R²s.” It might not have quite the rigor of academic papers, but it yields many of the same results. Even more importantly, “Unlike academics, Silver is unburdened by the constraining forces of peer review, turgid and esoteric disciplinary jargon, and the unwieldy format of academic manuscripts. He need not kowtow to past literature, offer exacting descriptions of his methods, or explain in tedious detail how his findings contribute to existing theory.”

That essay proved prescient because on April 1, 2014, Nate Silver scooped me (how cruel to do it on April Fool’s Day!).

Well, it wasn’t exactly Nate Silver. It was Walt Hickey, a writer for FiveThirtyEight. But Hickey’s article essentially yielded that same results as a paper I’d been trying to publish since 2011. In this post, I explain how social science’s dysfunctional system of peer-review and publication inhibits the dissemination of worthwhile findings using my article as a case study. (I also point out some of the value of our system).

For the record, I really like Walt Hickey’s writing and how he has consistently raised issues of sexism in film. I don’t think he stole my idea at all. I’m also proud of the paper I wrote with two undergraduate students. This post isn’t a case of sour grapes so much as a very common backstory that we rarely hear.

***

Case Study

In summer of 2010, I had read about the Bechdel Test, a test of gender representation in film conceived of by graphic artist Alison Bechdel. The test asks, “Do two or more named women talk to each other about something other than a man?” I had heard that most movies failed. As I rode my bike to work that day, I wondered whether the underrepresentation of women in film was due to audiences disliking movies featuring women or Hollywood under-funding Bechdel movies. I cooked up the idea to link a content analysis of whether movies pass the Bechdel Test with data on the movies’ box office performance, production budget, and critical appraisal. That fall and winter, two wonderful students, Melissa Lindquist and Julie Arnold, and I collected the data. In short, we found that Bechdel movies earn less at the box office, but it’s because they have smaller production budgets, not because audiences reject them. A simple study, but, I think, an interesting one.

In August 2011, we presented the paper at the American Sociological Association meeting in Las Vegas. Based on positive feedback from several people who know what they’re talking about, I shot high and sent it to one of our top two journals. Figure 1 summarizes what happened next. Some of the methodological suggestions from Top Two Journal proved helpful and I did some extensive new coding after a re-sample. However, the changes did not substantively alter the findings. Because of the critique that it was not of broad interest in sociology, I reframed it to connect with more macro-level theory about gender and submitted to Gender Journal. Gender Journal rejected it saying it didn’t tell us enough about gender in society as a whole and suggested that I aim for a more niche gender journal. To be fair, I might have done that and published it sooner, but I believed it would be of interest to a general sociology audience. I reframed it to be about larger social processes connected to gender and a gendered division of labor. I sent it to Top Five Journal and it was rejected.

Figure 1. A Brief History of An Article

At this point, I faced facts that it wasn’t cut out for a top tier journal and that the literature review needed an overhaul. I was probably a bit demoralized, too, and had other projects, not to mention a 3-3 teaching load. So, I worked on the revision here and there and didn’t send it back out until after the school year ended in June 2013. My bad. I sent it to Journal of Regional Association and they sent back a desk R&R asking that I revise it as a Research Note, cutting the length in half. Delighted by the opportunity, I did that work in a month. Despite having to cut it in half, the reject letter said it needed more theory.

At this stage, I felt at a loss for what kind of theory people wanted … or maybe what theory even means. I’m all for theory – sociology shouldn’t be an archipelago of unlinked empircs – and, like others, I worry about the loss of social theory as a subfield. At the same time, every article can’t do everything. Journals should evaluate theory and make sure that data and theory make sense together, but reviewers shouldn’t be selecting on whether their pet theory is used.

In Jan. 2014, I sent it to Sociological Inquiry (once again as a full article) where it would ultimately be published. While it was being reviewed, the Hickey piece came out. One reviewer for Sociological Inquiry recommended acceptance and in the Comments section wrote, “Nothing in particular.” Best. Reviewer. Ever. The two other reviewers were very helpful and suggested using Griswold’s “cultural diamond” and Acker’s “gendered organizations.” I believe that applying these literatures improved the manuscript greatly.

What It Says About Science

My point is not that I was treated badly. Rather, there is nothing abnormal about this story. Every academic has a story to tell like this one. My study wasn’t and isn’t the best one ever and I probably aimed a bit too high initially. But after the Top Two Journal Submission, the critiques were not methodological. All the critiques were about theoretical framing. How many competent empirical studies are being held up as the authors revise to find a framing that suits everybody’s fancy? How many of us get demoralized by the process and dump a worthwhile manuscript in a drawer? How many important findings does the world not know about right now because of a review process where the wheels turn too slowly?

By contrast, from initial data collection to publication took Hickey about a month (Personal Communication). The FiveThirtyEight model is scooping sociology because we’re too damn slow and obsessed with “theoretical framings.” Some efforts like Sociological Science are speeding that process up (30 days guaranteed!). And in the natural sciences, arXiv allows people to post working papers and then vote them up. Meanwhile, our predominant model is broken.

Where Science Wins

No offense to Hickey and FiveThirtyEight, but my paper with Melissa and Julie is better than his post in some ways. Our dataset is not a convenience sample. It is a population of the hundred most widely-distributed films for a decade. Our codings probably have higher reliability than what comes from a crowdsourced web site. And the literature illuminates the social meaning of the findings more fully.

But the findings are the same. On a friend’s comprehensive exams, one of the questions was, “What is one thing we have learned from multi-level modeling that we wouldn’t have known anyway?” There is a good answer to that question, but it’s limited. As a field, we need to consider how much all our fussiness over methods and theory is worth it. “The philosophers have only interpreted the world, in various ways; the point is to change it,” wrote one of our great theorists. It’s hard to change the world much if many of our empirical findings are still under review.

What do you think? What can we be doing as researchers, peer reviewers, and as a scientific community to address these issues?