When I teach social statistics, I often show students how small changes in measurement or analysis can make a big difference in the way we understand the world. Recently, I have been surprised by some anger and cynicism that comes up when we talk about this. Often at least one student will ask, “does it even matter if you can just rig the results to say whatever you want them to say?”

I can’t blame them. Controversy about manufactured disagreement on climate change, hoax studies, or the rise of fake news and “both side-ism” in our politics can make it seem like everyone is cooking the books to get results that make them happy. The social world is complicated, but it is our job to work through that complexity and map it out clearly, not to throw up our hands and say we can’t do anything about it. It’s like this optical illusion:

The shape isn’t just a circle or a square. We can’t even really say that it is both, because the real shape itself is complicated. But we can describe the way it is built to explain why it looks like a circle and a square from different angles. The same thing can happen when we talk about debates in social science.

A fun example of this popped up recently in the sociology of religion. In 2016, David Voas and Mark Chaves published an article in the American Journal of Sociology about how rates of religious commitment in the United States are slowly declining. In 2017, Landon Schnabel and Sean Bock published an article in Sociological Science responding to this conclusion, arguing that most of the religious decline was among moderate religious respondents—people with very strong religious commitments seemed to be holding steady. Just recently, both teams of authors have published additional comments about this debate (here and here), analyzing the same data from the General Social Survey.

So, who is right?

Unlike some recent headlines about this debate, the answer about religious decline isn’t just “maybe, maybe not.” Just like the circle/square illusion, we can show why these teams get different results with the same data.

Parallel Figures from Voas & Chaves (2018) and Schnabel & Bock (2018) (Click to Enlarge)

When we put the charts together, you can see how Voas and Chaves fit straight and smoothly curved lines to trends across waves in the GSS. This creates the downward-sloping pattern that fits their conclusions about slow religious decline over time. Schnabel and Bock don’t think a single straight line can accurately capture these trends, because the U.S. saw a unique peak in religious commitment that happened during the Regan years and may have receded more quickly. Their smoothing technique (LOESS smoothing) captures this peak and a quick decline afterwards, and doing so flattens out the rest of the trends after that period.

The most important lesson from these charts is that they don’t totally get rid of the ambiguity about religious change. Rather than just ending the debate or rehashing it endlessly, this work helps us see how it might be more helpful to ask different questions about the historical background of the case. I like this example because it shows us how disagreement among experts can be an invitation to dig into the details, rather than a sign we should just agree to disagree. Research methods matter, and sometimes they can help us more clearly explain why we see the world so differently.

Evan Stewart is an assistant professor of sociology at University of Massachusetts Boston. You can follow his work at his website, or on BlueSky.