RUTHFULLY YOURS

Jareau Hall breezed through high school in Syracuse, N.Y. Graduating in the top 20% of his class, he had been class president and a successful athlete, and he sang in gospel choir. He was actively recruited by Colgate University in rural New York, one of the nation’s top liberal-arts colleges.

None of Colgate’s recruiters mentioned to Mr. Hall that his combined math and verbal SAT scores were some 250 points below the class median—let alone that this would put him at great risk of academic difficulty.

Arriving at Colgate in 2002, he quickly found himself struggling in class, with far more rigorous coursework than he had ever faced. “Nobody told me what would be expected of me beforehand,” recalls Mr. Hall, now 28. “I really didn’t know what I was getting into. And it all made me feel as if I wasn’t smart enough.”

To make things worse, recalls Mr. Hall, “I was immediately stereotyped and put into a box because I was African-American. And that made it harder to perform…. There was a general feeling that all blacks on campus were there either because they were athletes or they came through a minority-recruitment program and might not really belong there.” Shaken by the experience, Mr. Hall dropped out after his freshman year. He eventually returned to Colgate and graduated in 2007.

There are, of course, a great many students who are admitted under affirmative action and go on to successful careers, just as there are a significant number of black and Hispanic students at elite schools who get in without any preference. But stories like Mr. Hall’s are both surprisingly common and seldom told. In fact, the majority of students admitted with large racial preferences struggle academically and often never come close to achieving their goals. At selective schools, more than 80% of blacks, and two-thirds of Hispanics, have received at least moderately large admissions preferences, according to our analysis of admissions data from several dozen selective schools—that is the equivalent of at least a 100-point SAT boost, and often much more.

For more than 40 years, the debate over affirmative action in admissions has focused on whether it amounts to unfair and unconstitutional reverse discrimination against whites (and now Asians). The implicit premise for most people on both sides has been that racial preferences bring only benefits and no costs, apart from the possible stigma of being deemed “affirmative-action admits,” to their black and Hispanic recipients. This premise was enough to make the two of us uncritical supporters of racial preferences until we began to examine the underlying facts.

Key to nurturing the myth that racial preferences can only help their recipients has been a strong norm among college administrators to play down both the size of preferences they use and the difficulties these students encounter down the road. This concealment has had the unfortunate effect of misleading students and shielding preference policies from close scrutiny.

But cracks of light have begun to leak through. There is now increasing evidence that students who receive large preferences of any kind—whether based on race, athletic ability, alumni connections or other considerations—experience some clear negative effects: Students end up with poor grades (usually in the bottom fifth of their class), lower graduation rates, extremely high attrition rates from science and engineering majors, substantial self-segregation on campus, lower self-esteem and far greater difficulty passing licensing tests (such as bar exams for lawyers).

The most encouraging part of this research is the parallel finding that these same students have dramatically better outcomes if they go to schools where their level of academic preparation is much closer to that of the median student. That is, black and Hispanic students—as well as the smaller numbers of preferentially admitted athletes and children of donors—excel when they avoid the problem of what has come to be called “mismatch.”

Jareau Hall’s experience is representative of scenes that play out every fall at selective schools across the country. Black and Hispanic high-school seniors are actually more likely than similar whites to aspire to careers in science and engineering (which, along with technology and math, make up the so-called STEM fields), as first demonstrated by Dartmouth psychologist Rogers Elliott in 1996, and since confirmed in other studies. Tens of thousands of minority students receive preferences to attend schools where they feel overwhelmed, especially in STEM classes. As a result, the studies have found, they switch from science courses and migrate to other fields that, if not actually easier, are at least graded less harshly and are less sequential in their teaching. The end result: Whites are seven times more likely than blacks to go on to get doctorates in STEM fields.

Mr. Elliott’s research was ignored in the widely heralded 1998 book “The Shape of the River,” in which former Princeton President William Bowen and former Harvard President Derek Bok argued that the effects of affirmative action were unambiguously good. But a few years later, University of Virginia psychologists Fred Smyth and John McArdle got access to the data set created by Messrs. Bowen and Bok, which included data on academic aspirations, admissions and performance for tens of thousands of students at 28 major universities. Using the same data, they showed that students receiving large preferences were nearly 80% more likely to complete STEM degrees if they avoided mismatch by going to a less-elite school.

Research on law schools by one of us (Richard Sander)—hotly disputed by some scholars when it was published in 2005 by the Stanford Law Review and now confirmed by economist Doug Williams—found that mismatch essentially doubled the rate at which blacks and Hispanics failed bar exams. Under existing preference policies, only one in three blacks entering law school graduates and passes the bar on his or her first attempt (compared with two in three whites). Simply by reducing mismatch, we could get this ratio up to one in two.

This same dynamic turned up when several leading educators wanted to find out why so few black students went on to become professors. Funded by the Council of Ivy League Presidents, sociologists Stephen Cole and the late Elinor Barber surveyed thousands of young African-American students entering a broad cross-section of selective schools. The 2003 Cole-Barber book, “Increasing Faculty Diversity,” concluded that large racial preferences and the ensuing mismatch led directly to lower grades and diminished intellectual self-confidence. They found that promising young black students who wanted to become professors abandoned their academic aspirations in droves, while similar black students who weren’t mismatched were far more likely to stay the course.

The work ties into a second major finding, that of social mismatch. The central legal justification for using race in university admissions is the need to produce a healthy learning environment by fostering diverse classroom viewpoints and cross-racial friendships. But it turns out that these effects are also heavily influenced by the presence of large preferences. Economics professor Peter Arcidiacono and his colleagues at Duke University found in a 2011 study that students were much more likely to become friends with classmates they saw as academically similar to themselves. Students with large preferences were more likely to self-segregate and find themselves socially isolated.

The reason wasn’t racism. At Duke University, for example, large numbers of whites and blacks formed friendships at the outset of college. But for those with large academic gaps, the friendships atrophied. Using their multischool results, Mr. Arcidiacono and his colleagues concluded that smaller preferences at the most selective schools would tend to increase both the likelihood and the number of cross-racial friendships at elite schools in general, despite declines in the numbers of black and Hispanic students at the most elite schools.

Interviews that we and our colleagues conducted with dozens of black and Hispanic administrators and former students revealed a striking theme: Almost all of them complained that blacks are stereotyped on campus as being weak students. It is, of course, not surprising that the large performance gaps on campus that highly correlate with race tend to foster—rather than undermine—racial stereotypes. Indeed, scholars at Harvard and University of California, Los Angeles have shown that students who are aware that they have received a racial admissions preference are more likely to think that they are being negatively stereotyped, and those students appear to do worse academically because of that perception.

Even as social scientists have transformed our understanding of affirmative action, universities don’t seem to be paying attention. Consider the University of California system, which since 1998 has been legally precluded (by Proposition 209) from considering race in admissions. Throughout the past 15 years—most recently in a brief submitted to the Supreme Court—university officials have denounced race neutrality and pointed to the substantial drop in freshman black and Hispanic students at the system’s two flagship schools, Berkeley and UCLA.

Yet race-neutrality has produced stunning benefits for minorities in the UC system as a whole, as shown in a data set that economists obtained from UC administrators. Black, American-Indian and Hispanic students made up 26% of all U.C. freshmen in 2010, up from 16% in 1997; the number of B.A.s earned by black and Hispanic students in four years rose 55% between 1995-97 and 2001-03, while the number with GPAs above 3.5 rose 63%.

What can be done about the problem of mismatch? Most obviously, we need dramatic improvements in elementary and secondary schools to narrow the racial gaps in academic achievement. According to the National Assessment of Educational Progress, the average black 12th-grader is on a par with the average white eighth-grader. That project will take decades.

Meanwhile, mismatched students can benefit from remedial academic support programs. Mr. Hall’s return to Colgate was successful, he recalls, in part because he made a point of consistently communicating with his professors and working closely with his guidance counselor. Mr. Hall ended up graduating with a 2.5 GPA overall and about a 3.1 in his major, African studies. Colgate is a good school, he says, but he now believes he would have preferred a college that was, among other things, “more diverse and in an urban area.”

But remedial programs can do only so much. And it seems clear that reform of the racial-preference regime will never come from universities. Nor will reform come from elected officials, who are so terrified of being attacked as racist that even Republican candidates who personally oppose racial preferences (as did two-thirds of respondents in a 2009 Quinnipiac survey) dare not make them an issue.

Racial-preference reform, in short, can come only from the Supreme Court—which, this past Wednesday, struggled to come to grips with its messy past rulings on the use of racial preferences. In evaluating whether the University of Texas denied equal protection of the laws to Abigail Fisher by taking race into account in admissions, the court is faced with such questions as how to find the presence of a “critical mass” of minority students without using numerical targets; whether race is the “determining factor” in any admissions decision; and whether surveys by university officials confirming their own assertions that campus minorities are socially isolated create a “compelling governmental interest” in admitting more blacks and Hispanics.

We badly need a simpler, more coherent and more workable set of rules about affirmative action. We have a few simple suggestions.

First, the court should mandate transparency about the actual operation of preferences. Applicants—and scholars, voters and policy makers—should be able to know exactly how much weight a university gives to the various factors used in admissions. Applicants who are offered admission should be given information about how students with their level of academic preparation typically fare at the college—so that prospective students can evaluate the danger of mismatch.

Second, racial preferences should not be permitted to exceed the size of a school’s socioeconomic preferences (which at most schools are now minuscule). That is, schools need to demonstrate that if they wish to use race, it is as a supplement to a fundamentally more honest measure of disadvantage. This would also create incentives to seek more socioeconomic diversity. Of course, as schools create more genuine diversity, they also need to make special efforts to help less privileged students of all races succeed academically and navigate the new social environment they will encounter at college.

Third, schools should not be permitted to use race-based scholarships. Genuine need can be fully met through need-based scholarships; the race-based kind simply foster the sort of zero-sum competition that now causes American law schools to give four times as much grant aid to rich blacks as to poor whites, as one of us (Richard Sander) found in a 2011 study for the University of Denver Law Review.

These measures won’t solve all the problems of affirmative action; neither will they completely overturn the idea of race-consciousness in admissions. But they will set us on the path to more honest policies and inquiry.

—Mr. Taylor, a legal journalist and author, and Mr. Sander, a UCLA law professor and economist, are the authors of “Mismatch: How Affirmative Action Hurts Students It’s Intended to Help, and Why Universities Won’t Admit It,” published this week.A version of this article appeared October 12, 2012, on page C1 in the U.S. edition of The Wall Street Journal, with the headline: The Unraveling of Affirmative Action.