Improbable scholars, inappropriate conclusions

School by

Article Highlights

  • Improbable Scholars lacks the very humility and appreciation for nuance that the educators it chronicles possess.

    Tweet This

  • The plural of anecdote is not data. McShane reviews Improbable Scholars via @educationnext.

    Tweet This

  • I don’t think that we can draw any broader policy recommendations from Improbable Scholars.

    Tweet This

Improbable Scholars: The Rebirth of a Great American School System and a Strategy for America’s Schools
By David L. Kirp
Oxford University Press, 2013, $24.95; 272 pages

As reviewed by Michael McShane

The book jacket of David Kirp’s Improbable Scholars describes it as a “playbook—not a prayer book—for reform that will dramatically change our approach to reviving public education.” Kirp sees the beating-the-odds story of Union City, New Jersey—a high-poverty school district that turns out high test scores and graduation rates—as a challenge to the agenda of “education reformers.”

Union City, Kirp argues, is successful because it fosters a culture of continuous improvement of existing staff and schools, utilizes bilingual instruction, and guarantees high quality pre-school—eschewing high stakes accountability, Teach for America teachers, and school choice. He profiles in depth the efforts of school leaders to instill a culture of abrazos—caring—while still pushing students to achieve to their fullest potential.

Unfortunately, as the saying goes, the plural of anecdote is not data. Kirp selected a district that was already doing well, and then tried to back out practices that made it succeed where other fail. But what about other districts that are doing the same things as Union City but with poor results? What about districts that are doing the opposite of Union City and succeeding? Is Union City an outlier because of some other variable that this single case study fails to take into account? Is Union City the rule or the exception?

Kirp’s analysis lacks the power to answer these questions because it violates a central tenet of social science research—it selects on the dependent variable. Though he adds three more cities in his conclusion, they cannot shed any additional light on causality because of the way they were chosen.

I have zero doubt that the Union City school district is a great institution. The teachers Kirp profiles are rock stars and heroes and should be applauded for their work. I just don’t think that we can draw any broader policy recommendations from this book.

But even beyond that, Kirp’s clear political agenda leads to distracting asides. Early in the book, when describing the history and context of Union City, he sneaks in two paragraphs about how it is wrong to oppose the DREAM act, before settling back in to describe the number of students that receive subsidized lunches. A page later Kirp asserts that “you won’t find any Teach for America recruits on a mission to reinvent urban education” at Washington Elementary School, unnecessarily knocking folks who have nothing to do with this story. The author takes every chance he gets to slight New Jersey Governor Chris Christie, whom he says “made a name for himself by beating up on educators.” These comments are gratuitous and unnecessary.

Also, Kirp says some things that are just wrong.

On page 212, he inaccurately summarizes the research on school vouchers. First, he uses a 2002 GAO study to say that students who receive vouchers fare no better than comparable public school students, even though a veritable mountain of evidence to the contrary has been published since then. Also, he describes a 2012 study of the Milwaukee Parental Choice Program by John Witte and Patrick Wolf as the “best-designed” study of school vouchers. That just isn’t true. The Witte/Wolf study used a matched comparison design, following a cohort of students that participated in the program and comparing them to a similar set of students in the Milwaukee Public Schools. But in any evaluation where there is fear of selection bias, matched comparison designs are inferior to randomized control trials. There is a huge risk of selection bias in school voucher research, so while the Witte/Wolf study is useful, and used the best possible method given the limitations of the program as it was implemented, it is not the “best-designed” study of school vouchers. What’s more, there have been 10 random assignment studies of school vouchers (the best design possible) and 9 of them have found positive results for some or all participating students.

Kirp also says on page 6 that California is a “textbook example” of white voters refusing to tax themselves to pay for schools that primarily educate poor and minority students. I’m not sure how to square that with the fact that California passed Proposition 30 by a 55/45 margin in November, increasing both income and sales taxes to raise nearly $7 billion per year for schools, minority children and all.

Those complaints aside, what is most disheartening about this book is that it lacks the very humility and appreciation for nuance that the educators it chronicles possess. The assurance with which Kirp offers his policy recommendations, going so far as to say that they’d look familiar to “any educator with a pulse,” stands in stark contrast to the approach of the teachers and school leaders who adapt to the needs of their students. We can all learn from their example.

Michael McShane is a research fellow at AEI and co-author of President Obama and Education Reform: The Personal and the Political.

Also Visit
AEIdeas Blog The American Magazine
About the Author


Michael Q.

What's new on AEI

Love people, not pleasure
image Oval Office lacks resolve on Ukraine
image Middle East Morass: A public opinion rundown of Iraq, Iran, and more
image Verizon's Inspire Her Mind ad and the facts they didn't tell you
AEI on Facebook
Events Calendar
  • 21
  • 22
  • 23
  • 24
  • 25
Monday, July 21, 2014 | 9:15 a.m. – 11:30 a.m.
Closing the gaps in health outcomes: Alternative paths forward

Please join us for a broader exploration of targeted interventions that provide real promise for reducing health disparities, limiting or delaying the onset of chronic health conditions, and improving the performance of the US health care system.

Monday, July 21, 2014 | 4:00 p.m. – 5:30 p.m.
Comprehending comprehensive universities

Join us for a panel discussion that seeks to comprehend the comprehensives and to determine the role these schools play in the nation’s college completion agenda.

Tuesday, July 22, 2014 | 8:50 a.m. – 12:00 p.m.
Who governs the Internet? A conversation on securing the multistakeholder process

Please join AEI’s Center for Internet, Communications, and Technology Policy for a conference to address key steps we can take, as members of the global community, to maintain a free Internet.

Event Registration is Closed
Thursday, July 24, 2014 | 9:00 a.m. – 10:00 a.m.
Expanding opportunity in America: A conversation with House Budget Committee Chairman Paul Ryan

Please join us as House Budget Committee Chairman Paul Ryan (R-WI) unveils a new set of policy reforms aimed at reducing poverty and increasing upward mobility throughout America.

Thursday, July 24, 2014 | 6:00 p.m. – 7:15 p.m.
Is it time to end the Export-Import Bank?

We welcome you to join us at AEI as POLITICO’s Ben White moderates a lively debate between Tim Carney, one of the bank’s fiercest critics, and Tony Fratto, one of the agency’s staunchest defenders.

No events scheduled this day.
No events scheduled this day.
No events scheduled this day.
No events scheduled this day.
No events scheduled this day.