Perspectives on Standardized Testing
Download Q&A
Perspectives from:
Richard H. Sander, Jesse Dukeminier Professor of Law, UCLA Law School
Bob Schaeffer, Executive Director for FairTest: National Center for Fair & Open Testing
1. What is the purpose of the SAT/ACT exam and do the exams fulfill that purpose?
Sander: The SAT is the older of the two exams, and was introduced in the early 20th century as a source of more objective information about the academic ability and preparation of high school students applying to college. Many elite colleges at that time disproportionately favored the graduates of New England prep schools; the SAT led to more economic and geographic diversity at these schools. Today, the SAT and ACT are still the single best source college admissions officers have for objective information about a student’s ability and preparation to succeed in college.
Schaeffer: For decades exam-manufacturers have promoted the ACT/SAT as a helpful admissions tool primarily because of its moderate value in forecasting first-year undergraduate academic performance. Research by both the testing industry and independent scholars, however, demonstrates that the prediction value is weak: test scores account for less than 20% of differences among freshman student grades. Moreover, many studies have found that high school grades are better predictors. In addition, admissions professionals increasingly recognize that the tests do not do a good job of predicting multi-year grades or graduation, outcomes that are much more important.
2. Do the exams predict college readiness and success better than other measures such as grades?
Sander:
Over the years, there have been hundreds (perhaps thousands) of “validity” studies that examine how well various metrics predict college performance. Inevitably, the results of these studies vary, but the overwhelming majority of the studies find that SAT/ACT exams are the best predictors of a wide variety of academic outcomes, such as freshman year college grades, GPAs at college graduation, and graduation itself (versus dropping out).
One of the most thorough of these studies was completed in January 2020 by a special University of California faculty task force asked by the chair of the UC Academic Senate to evaluate the proper role of the SAT and ACT in UC admissions.[1] The task force was diverse both in its ideological makeup and in the disciplines of its members, but it unanimously agreed on many basic factual findings about standardized testing. For example, in examining outcomes for nearly one hundred thousand UC students, it found a correlation of SAT and first-year college grades of .46, higher than the correlation between high school grades and first-year college grades (.39). Students with low SATs had a 35% attrition rate after one year at UC; students with high SATs had only a 3% attrition rate.
A .46 correlation is substantial, but it’s important to note that this understates the actual predictive utility of the exam, because of the “restriction of range”problem. Since colleges generally use the SAT to select students, the students at a given campus tend to have relatively similar SAT scores, which then reduces its utility in predicting outcomes among that group of students. When one adjusts for this issue, SAT/ACT correlations with college grades are significantly higher. When one also uses test scores to predict, not all college freshman GPAs (since college freshmen take widely varying mixes of classes), but GPAs of freshmen taking a common set of courses, the correlation rises to about .68.3. Of course, the fact that SAT/ACT scores generally predict college performance better than high school grades is not (at all) a reason to ignore high school grades. Admissions officers should (and generally do) use all types of information that help predict performance. Indexes that combine test scores and grades predict performance better than either scores or grades in isolation.
[1] Report of the UC Academic Council Standardized Testing Task Force (2020).
[2] Id. at 21.
[3] Paul Sackett and Nathan Kuncel, Eight Myths About Standardized Testing, chapter one of Buckley et al, Measuring Success: Testing, Grades and the Future of College Admissions (Johns Hopkins, 2018), p.17. Sackett and Kuncel also provide extensive evidence about the high correlations between test scores and longer-term achievements of students beyond college.
Schaeffer: Definitely no, according to studies by many colleges and even the testmakers themselves. An applicant's high school record (grades, course rigor and other factors) are much stronger predictors.
3. What factors lead to a student having a high score?
Sander: High scores reflect both strong academic preparation and cognitive ability, and each of these is influenced by an enormous array of factors. The quality of one’s elementary and secondary education is important, but so are many personality and environmental factors – how much is a given teenager (and the teenager’s friends) interested in academic subjects and books? What is he or she passionate about? The early childhood environment is also very influential in shaping later academic outcomes: how many books young children are exposed to; how many different words their parents use; even how many hours a day a television is on in a child’s home, and whether the child was born prematurely or with low birthweight.
Schaeffer: Admissions exam scores reflect many factors, including a student's learning opportunities, test-taking skills, and -- to some extent -- academic training. Affluent parents can buy their children the test-prep equivalent of steroids, which boost their scores without necessarily making them better students.
4. Can a candidate materially improve one's score in a short period of time through prep courses?
Sander:
The benefits of prep courses are quite modest. Naturally, the companies that offer prep courses make extravagant claims about their benefits, and of course an occasional student who bombs the SAT will take a prep course and then get a much higher score. But the careful, independent studies of this question that are based on large samples consistently find only small average improvements. A meta-analysis by D.C. Briggs, for the National Association of College Admissions Counselors,[1] found that test-prep courses produced, on average, a 15-20 point improvement in SAT math scores, and an 8-10 point improvement on SAT verbal scores – very small gains that would have only a marginal effect on student admissions chances. An exceptionally careful study by Connally, Sackett, and Waters found that students taking test prep courses gained 14-15 points on their SAT math score (relative to their PSAT score), and 4-10 points on the SAT verbal.[2].
[1] Briggs, D.C., Preparation for College Admissions Exams (published by the National Association of College Admisssions Counselors).
[2] Connally, Sackett, and Waters, “Balancing treatment and control groups in quasi-experiments: An introduction to propensity scoring,” 66 Personnel Psychology 407 (2013).
Schaeffer: As the test-makers themselves now admit, good coaching courses can raise SAT scores by 150 points or more and ACT scores by several points. Sponsors of both exams point students to test-prep companies with whom they are affiliated.
5. Are low-income students at a disadvantage with respect to the exams?
Sander:
It is certainly true that low-income students, on average, have lower SAT/ACT scores than high-income students, but this does not mean that these tests “disadvantage” or “discriminate against” them. To understand this issue, one needs to keep three points in mind.
First, the correlation between income (or, more broadly, socioeconomic status) and SAT scores is quite low – between about .15 and .20. There is substantially more variation in SAT scores within any socioeconomic group than between groups. Microdata from the College Board shows that in 2012, for example, 35% of students with low SES (socio-economic status) had combined verbal and math scores of over 1000, while 35% of students with high SES had combined verbal and math scores below 1000. In other words, there are vast numbers of high-scoring, low-income students, and vast numbers of low-scoring, high-income students.
Second, SAT/ACT scores predict college performance as well for low-income students as for high-income students. For example, in the sophisticated UC study cited above, the correlation of SAT scores and freshman grades was .41 for students with family incomes under $30,000, and .35 for students with family incomes over $120,000.[1]
Third, and most importantly, SAT/ACT scores do not under-predict college success for low-income students. If low-income students were getting better college grades, or having higher graduation rates, than high-income students with similar scores, then that would be evidence that the tests were not adequately capturing their actual level of academic preparation. But there is no evidence of such under-prediction; if anything, the SAT/ACT scores slightly over-predict low-income students’ college success (but this is a very slight effect).
[1] See supra note 1, at 21.
Schaeffer: There is a strong correlation between ACT/SAT scores and measures of socio-economic status, including family income, wealth and parental education.
6. Are racial minorities, irrespective of income, at a disadvantage respect to the exams?
Sander:
My answer to question 5 largely applies here, too. The SAT and ACT do not disadvantage or discriminate against minorities, because they do not underpredict minority performance in college, and the relationship between test scores and college outcomes (graduation, grades, etc.) is as strong or stronger for Blacks and Hispanics as it is for Whites and Asian-Americans.
There is a very large racial test score gap, but it accurately reflects very serious differences in the average performance of students of different races in high school. The U.S. Department of Education undertakes large, intensive, and careful studies each year of academic achievement for tens of thousands of K-12 students. This National Assessment of Educational Performance (NAEP) finds that Black high school students, on average, lag four years behind White high school students in academic achievement. Hispanic students, on average, lag two-to-three years behind Anglo (non-Hispanic white) students.[1] The reasons for these gaps are not “race” per se, but differences in schooling, neighborhood environment, parenting practices, and socioeconomic status that each correlate to some degree with race.[2]
[1] See National Assessment of Educational Progress, Trends in Academic Progress: 2012, pp. 17-18. More recent data is available at NAEP’s website, https://nces.ed.gov/nationsreportcard/ ; the patterns have not changed much since 2012, but the 2012 report is particularly helpful in reporting the results clearly.
[2] Roland Fryer and Steven Levitt, “Understanding the Black-White Test Score Gap in the First Two Years of School,” The Review of Economics and Statistics, May 2004, pp. 447-464.
Schaeffer: In general, yes. African-American, Latino, Native American and recent Asian immigrant groups score significantly lower than their white counterparts.
7. Should the score be indexed for family income or other factors?
Sander: Indexing SAT/ACT scores is a bad idea because, as noted above, the scores are unbiased predictors of college performance. If admissions officers wish to give special consideration (as I think they should) to someone’s socioeconomic background, they should be transparent about this, and not hide the preference in “SES-normed” scores. They should also monitor the use of preferences carefully to avoid setting students up for failure by placing them in academic environments where they will have difficulty competing.
Schaeffer: Applying an arbitrary correction to flawed test scores would not magically transform them into better predictors of undergraduate success. The truth is that ACT/SAT results are not needed to make accurate and fair admissions decisions.
8. Why have so many schools gone SAT/ACT optional or SAT/ACT-blind (excluding the temporary COVID impact)?
Sander:
There seem to be two reasons. Some smaller schools – often schools that were struggling financially – started this trend because they felt it gave them a competitive edge, and would attract students who had high grades but low test scores.
More recently, many larger schools have taken this route because they want to appease “progressive” and “woke” forces in the media, among students, and in other constituencies. The UC Regents, for example, ignored the careful, accurate, 200-page report prepared by UC’s faculty task force, and eliminated use of the SAT on politically-motivated, fact-free grounds. There are, unfortunately, many demagogues who make utterly false claims about racial bias in standardized tests – people who seem to believe that if one gets rid of the evidence about racial differences in academic preparation, those differences will no longer matter.
Schaeffer: Even before the pandemic, 1,070 accredited bachelor-degree granting institutions (46% of all such schools) had ACT/SAT optional policies. Another 700 schools joined them in 2021 when COVID shut down many testing centers. Colleges and universities that eliminated test-score requirements generally saw more applicants, better academically qualified applicants and more diversity of all types.
9. Will it be more challenging for admissions officers to differentiate among students without an objective marker (i.e., given high schools have varying levels of academic rigor and grade inflation)?
Sander:
Yes, of course. In fact, the predictive power of high school grades has declined in recent years,[1] and experts believe that the utility of high school grades is in jeopardy from continuing grade inflation in high schools.[2] As more colleges dispense with test scores, the importance of grades – and the pressure from students and parents for more grade inflation – will only increase. Admissions officers may then tend to place more emphasis on things like letters of recommendation and personal essays, where affluent students really do have an unfair advantage.
[1] UC study, supra note 1, at 19.
[2] Sackett and Kuncel, supra note 3, at 25.
Schaeffer: No -- admissions offices have quickly adjusted to the new normal of test-optional admissions. They recognize that there is more than sufficient information about an applicant’s academic potential in other parts of her file to make high quality admissions decisions. Many also know that ACT/SAT scores are not "objective" -- except for scoring-by-machine, all parts of the testing process involve subjective human decision from choosing which topics are covered, to writing questions, to designing both correct and wrong answers, to determining how scores are tabulated. Despite the differences among high schools in course content and grading, grades remain better predictors than test scores.
10. What are the risks of removing the exams from admissions considerations?
Sander:
What we are really talking about is making college admissions more random and arbitrary, and less connected to student levels of academic achievement. There are three risks here.
First: when we disregard the information in high test scores, we reduce the ability of colleges to gather together a class of high-achieving students. Concentrations of talent spur students on to excellence, and lead to extraordinary accomplishments that benefit the students, the academic environment of universities, and the larger society.
Second: when we disregard the information in low test scores, we will more often admit students to colleges for which the student lacks adequate preparation. Colleges tend not to do a good job of mentoring and nurturing their weakest students; those students are instead likely to be mismatched, to struggle academically, and to get much less out of their education than they would if they attended a school where other students have similar levels of academic achievement, and where instruction is therefore pitched at a level that allows the student to learn and flourish.
Third: society is fairer, and provides appropriate incentives for hard work, when we allow students to demonstrate their academic achievement (through, for example, standardized tests) and to be judged based on their achievement rather than on characteristics, like race, which they cannot change.
Schaeffer: A major risk is that a few institutions might try to substitute other biased or otherwise flawed measures, such as the number of Advanced Placement courses an applicant has taken, for ACT/SAT results. However, the 50-year track record of test-optional admissions at institutions such as Bowdoin and Bates indicates that responsible admissions officers can avoid these potential pitfalls.
11. Will the SAT/ACT still be utilized as an admissions tool in 5 years? If so, will it be used differently than in the past? If not, what will replace it?
Sander: Since the movement against the SAT and ACT is fundamentally irrational and dishonest, one can hope that it is simply a fad which will run its course. On the other hand, if wokeness continues to gather strength, then the SAT will indeed go extinct. The next target of the woke crowd will be grades themselves; these too will be denounced as unfair and discriminatory.
Schaeffer: FairTest projects that the ACT/SAT will not disappear in the mid-range future. Rather, they are more likely to play a lesser role in the admissions process at most institutions, perhaps more like that of the SAT Subject Tests and SAT Essay until both were recently withdrawn from the marketplace. Nearly 1,800 colleges and universities are ACT/SAT-optional or test-blind/score-free for fall 2022 applicants, and more than half of all schools have committed to extending those policies at least through fall 2023 and often years beyond.
12. Should the SAT/ACT exams continue to be used as admissions tools?
Sander: Of course. While we should continue to search for new kinds of diagnostic tools that can help us assess achievement in different and innovative ways, we should not discard tools of proven value until we have an alternative that, based on real evidence, is even better.
Schaeffer: That's a decision for admissions offices to make based on their own institutional research and mission. However, FairTest believes that schools should not require ACT/SAT submission unless they can produce strong evidence that test scores positively contribute to a fair and accurate evaluation of all applicants.