Cross-posted at Montclair SocioBlog.
Is the SAT biased? If so, against who is it biased?
It has long been part of the leftist creed that the SAT and other standardized tests are biased against the culturally disadvantaged – racial minorities, the poor, etc. Those kids may be just as academically capable as more privileged kids, but the tests don’t show it.
But maybe SATs are biased against privileged kids. That’s the implication in a blog post by Greg Mankiw. Mankiw is not a liberal. In the Bush-Cheney first term, he was the head of the Council of Economic Advisors. He is also a Harvard professor and the author of a best-selling economics text book. Back in May he had a blog post called “A Regression I’d Like to See.” If tests are biased in the way liberals say they are, says Mankiw, let’s regress GPA on SAT scores and family income. The correlation with family income should be negative.
…a lower-income student should do better in college, holding reported SAT score constant, because he managed to get that SAT score without all those extra benefits.
In fact, the regression had been done, and Mankiw added this update:
Todd Stinebrickner, an economist at The University of Western Ontario, emails me this comment:
“Regardless, within the income groups we examine, students from higher income backgrounds have significantly higher grades throughout college conditional on college entrance exam . . . scores.” [Mankiw added the boldface]
What this means is that if you are a college admissions officer trying to identify the students who will do best in college, as measured by grades, you would give positive rather than negative weight on family income.
Not to give positive weight to income, therefore, is bias against those with higher incomes.
To see what Mankiw means, look at some made-up data on two groups. To keep things civil, I’m just going to call them Group One and Group Two. (You might imagine them as White and Black, Richer and Poorer, or whatever your preferred categories of injustice are. I’m sticking with One and Two.) Following Mankiw, we regress GPA on SAT scores. That is, we use SAT scores as our predictor and we measure how well they predict students’ performance in college (their GPA).
In both groups, the higher the SAT, the higher the GPA. As the regression line shows, the test is a good predictor of performance. But you can also see that the Group One students are higher on both. If we put the two groups together we get this.
Just as Mankiw says, if you’re a college admissions director and you want the students who do best, at any level of SAT score, you should give preference to Group One. For example, look at all the students who scored 500 on the SAT (i.e., holding SAT constant at 500). The Group One kids got better grades than did the Group Two kids. So just using the SATs, without taking the Group factor (e..g., income ) into account, biases things against Group One. The Group One students can complain: “the SAT underestimates our abilities, so the SAT is biased against us.”
Case closed? Not yet. I hesitate to go up against an academic superstar like Mankiw, and I don’t want to insult him (I’ll leave that to Paul Krugman). But there are two ways to regress the data. So there’s another regression, maybe one that Mankiw does not want to see.
What happens if we take the same data and regress SAT scores on GPA? Now GPA is our predictor variable. In effect, we’re using it as an indicator of how smart the student really is, the same way we used the SAT in the first graph.
Let’s hold GPA constant at 3.0. The Group One students at that GPA have, on average, higher SAT scores. So the Group Two students can legitimately say, “We’re just as smart as the Group One kids; we have the same GPA. But the SAT gives the impression that we’re less smart. So the SAT is biased against us.”
So where are we?
- The test makers say that it’s a good test – it predicts who will do well in college.
- The Group One students say the test is biased against them.
- The Group Two students say the test is biased against them.
And they all are right.
————————
Huge hat tip to my brother, S.A. Livingston. He told me of this idea (it dates back to a paper from the1970s by Nancy Cole) and provided the made-up data to illustrate it. He also suggested these lines from Gilbert and Sullivan:
And you’ll allow, as I expect
That they are right to so object
And I am right, and you are right
And everything is quite correct.
Comments 23
John — July 8, 2011
I don't like using "family income" as the independent variable. It should be "family worth", which (should) include expected familial income, as well as assets.
Someone who owns their home that has a minimum wage job is orders of magnitude wealthier than someone who doesn't own their home with the same job.
Reise Ohne Ende — July 8, 2011
The two biggest issues I see in this analysis are:
1. While the SAT is standardized across the nation, GPAs are most certainly not. I believe that it is, on average, MUCH easier to earn a 4.0 GPA in an impoverished, dangerous school than in a better-funded, safer school, when taking into consideration only the difficulty of the material required to achieve high scores. So, while I don't think the SAT is indicative of intelligence, I think GPA is an even more useless measure.
2. Who says admissions officers should make their judgments on who will perform best in college? Why not who stands to benefit most, who will contribute most to the community, who will add a diversity of personalities and backgrounds, etc.?
guest — July 8, 2011
"Culturally disadvantaged." Ouch.
Liz Scott — July 8, 2011
i could have told you that without all the other tests and what not.
where to start...
Most of the professors i have say that every hour you spend in class you have to spend you should spend at least 2 studying (reading, writing, doing homework and so on). Also, depending how adept you are at a subject you may have to make that 3 or even 4 hours of study time. so, in a 12 credit semester you have to put in at least 24 hours of study time plus the 12 hours that you are in class. if you are a rich kid and and your parents are paying your rent or if you are really lucky they are paying your tuition that is it, you only have to put in your 36 hours and the rest of the time you can do whatever. if you are like me and you have to pay your own way then along with that 36 hours of school time you have 40 hours of work time. thus, in reality i have two full time jobs.
so yeah, rich kids are going to do better. i don't know how many times i have been way to wore out from work to study or half ass a paper because i dont have the time
WG — July 8, 2011
A huge assumption that a high GPA = success.
Anonymous — July 8, 2011
Except with this hypothetical data, group just as good a predictor of grades as SAT score is. If I know only the group of the student, I can estimate their grades (3.0 for group 1, and 2.5 for group 2), and over the 38 students, the total of the square of the error is 3.5GPA^2. The largest error is .5 GPA.
If I only know the SAT score, the minimum total square of the error is ~3.36 GPA^2. The largest error is just over .5 GPA.
Knowing only SAT score OR GPA, I can guess which group the student belongs to almost 75% of the time.
socialist — July 9, 2011
Both the SAT and college might be biased against some groups. That alternative fits the data just as well as Mankiw's claim. Mankiw simply assumes that college is bias free.
Feature X in the SAT bias the test against group A.
Feature Y in the college environment impedes the performance of group A.
X and Y might be one and the same: receiving lower SAT causes lower academic self-esteem causes worse performance.
Tamara — July 9, 2011
Hi, generally I love this blog, but there are some problems with the statistical analysis in this post. First, if you step back and look at the data in each graph for a moment, you can see that the regression line (for any traditional choice of how to regress a line) must be at a 45 degree angle (aka along a line connecting the upper righthand and lower lefthand corners) since the data is all symmetric about the 45 degree angle for the choices of axes given. Putting in the correct regression lines would make the Group 1 and Group 2 lines exactly the same, and there is nothing more to be said.
In the previous paragraph, we supposed that the data were correct and showed that the Group 1 and Group 2 lines must then be incorrect. Now let's suppose for a moment that the data points are incorrect, but the Group 1 and Group 2 lines are correct. E.g., we somehow experimentally verified these lines, so we will ignore the data points. In this case, recall that when you exchange the vertical and horizontal axes of a plot any lines in the plot should be flipped across the 45 degree line. This fact is generally true for two-dimensional plots. However, in the figures above, the two lines are not flipped across the 45 degree line when you go between one figure and the other. In fact, the lines plotted in both figures above are in exactly the same position even when the axes are reversed. So these plots cannot both be correct. Another way to see the problem is to note that the point (300,2.25) is on the red line in the first figure but (2.25,300) is definitely not on the red line in the second figure. Again, we see that the same lines are not being plotted across figures.
To see why the mistake pointed out in the preceding paragraph leads to a wrong analysis, note that you can condition on SAT in the first plot by forming a horizontal yellow ellipse (in this plot, you currently form a vertical yellow ellipse). Try doing this at a GPA of 3 (exactly the case in the second plot). Then the red line will, by extrapolation, hit 600 points for at GPA 3, but the blue line hits just 500. That is, Group 2 has a higher GPA for a given SAT---a consequence of the parallel linear model and consistent with the fact that Group 1 has a higher SAT for a given GPA.
Conclusion: The analysis is flawed. Actually, just a single person is "right" under the set of assumptions given.
Discussion: In real life, there may be many other complications. The GPA vs. SAT curve may not be linear. The lines for different classes, races, etc---if they exist---may not have the same slope (clearly, it is assumed above that they do). So I am merely pointing out an issue with the current post and not making any larger statement about the relation between class, GPA, and SAT.
Tomato Sauce — July 12, 2011
Your analysis in the second part is just wrong.
If you regress according to GPA (if the first graph is correct) you get that the Group 1 gets poorer SAT scores than Group 2 and NOT vis versa as you wrote. Regression cares not what is the "x-axis" and what is the "y-axis" so the second graph should just be the first graph flipped on its side, which it is clearly not (look for example the red line (group 2) at GPA=3, in the first graph it corresponds to SAT=450 while in the second GPA=600, clearly not the same data!). You have used to completely different data sets for the two graphs!
Tomato Sauce — July 12, 2011
Ooops... Tamara has already said what I wanted to say in a much better and more complete way, please look at Tamara's comment, she is exactly right in everything she says.
links for 2011-08-17 | Glenn Friesen — August 17, 2011
[...] SATs, GPAs, and Bias » Sociological Images Is the SAT biased? If so, against who is it biased? [...]
cheap bras — January 29, 2012
Amazing write-up! This could aid plenty of people find out more about this particular issue. Are you keen to integrate video clips coupled with these? It would absolutely help out. Your conclusion was spot on and thanks to you; I probably won’t have to describe everything to my pals. I can simply direct them here!