Improbable scholars, inappropriate conclusions

School by Shutterstock.com

Article Highlights

  • Improbable Scholars lacks the very humility and appreciation for nuance that the educators it chronicles possess.

    Tweet This

  • The plural of anecdote is not data. McShane reviews Improbable Scholars via @educationnext.

    Tweet This

  • I don’t think that we can draw any broader policy recommendations from Improbable Scholars.

    Tweet This

Improbable Scholars: The Rebirth of a Great American School System and a Strategy for America’s Schools
By David L. Kirp
Oxford University Press, 2013, $24.95; 272 pages

As reviewed by Michael McShane

The book jacket of David Kirp’s Improbable Scholars describes it as a “playbook—not a prayer book—for reform that will dramatically change our approach to reviving public education.” Kirp sees the beating-the-odds story of Union City, New Jersey—a high-poverty school district that turns out high test scores and graduation rates—as a challenge to the agenda of “education reformers.”

Union City, Kirp argues, is successful because it fosters a culture of continuous improvement of existing staff and schools, utilizes bilingual instruction, and guarantees high quality pre-school—eschewing high stakes accountability, Teach for America teachers, and school choice. He profiles in depth the efforts of school leaders to instill a culture of abrazos—caring—while still pushing students to achieve to their fullest potential.

Unfortunately, as the saying goes, the plural of anecdote is not data. Kirp selected a district that was already doing well, and then tried to back out practices that made it succeed where other fail. But what about other districts that are doing the same things as Union City but with poor results? What about districts that are doing the opposite of Union City and succeeding? Is Union City an outlier because of some other variable that this single case study fails to take into account? Is Union City the rule or the exception?

Kirp’s analysis lacks the power to answer these questions because it violates a central tenet of social science research—it selects on the dependent variable. Though he adds three more cities in his conclusion, they cannot shed any additional light on causality because of the way they were chosen.

I have zero doubt that the Union City school district is a great institution. The teachers Kirp profiles are rock stars and heroes and should be applauded for their work. I just don’t think that we can draw any broader policy recommendations from this book.

But even beyond that, Kirp’s clear political agenda leads to distracting asides. Early in the book, when describing the history and context of Union City, he sneaks in two paragraphs about how it is wrong to oppose the DREAM act, before settling back in to describe the number of students that receive subsidized lunches. A page later Kirp asserts that “you won’t find any Teach for America recruits on a mission to reinvent urban education” at Washington Elementary School, unnecessarily knocking folks who have nothing to do with this story. The author takes every chance he gets to slight New Jersey Governor Chris Christie, whom he says “made a name for himself by beating up on educators.” These comments are gratuitous and unnecessary.

Also, Kirp says some things that are just wrong.

On page 212, he inaccurately summarizes the research on school vouchers. First, he uses a 2002 GAO study to say that students who receive vouchers fare no better than comparable public school students, even though a veritable mountain of evidence to the contrary has been published since then. Also, he describes a 2012 study of the Milwaukee Parental Choice Program by John Witte and Patrick Wolf as the “best-designed” study of school vouchers. That just isn’t true. The Witte/Wolf study used a matched comparison design, following a cohort of students that participated in the program and comparing them to a similar set of students in the Milwaukee Public Schools. But in any evaluation where there is fear of selection bias, matched comparison designs are inferior to randomized control trials. There is a huge risk of selection bias in school voucher research, so while the Witte/Wolf study is useful, and used the best possible method given the limitations of the program as it was implemented, it is not the “best-designed” study of school vouchers. What’s more, there have been 10 random assignment studies of school vouchers (the best design possible) and 9 of them have found positive results for some or all participating students.

Kirp also says on page 6 that California is a “textbook example” of white voters refusing to tax themselves to pay for schools that primarily educate poor and minority students. I’m not sure how to square that with the fact that California passed Proposition 30 by a 55/45 margin in November, increasing both income and sales taxes to raise nearly $7 billion per year for schools, minority children and all.

Those complaints aside, what is most disheartening about this book is that it lacks the very humility and appreciation for nuance that the educators it chronicles possess. The assurance with which Kirp offers his policy recommendations, going so far as to say that they’d look familiar to “any educator with a pulse,” stands in stark contrast to the approach of the teachers and school leaders who adapt to the needs of their students. We can all learn from their example.

Michael McShane is a research fellow at AEI and co-author of President Obama and Education Reform: The Personal and the Political.

Also Visit
AEIdeas Blog The American Magazine
About the Author

 

Michael Q.
McShane

What's new on AEI

Defeating ISIS: AEI experts weigh-in before the president’s address on Wednesday
image Degrading, defeating, and destroying the Islamic State
image Wealth Building Home Loan: Building wealth through homeownership and retirement savings
image The $3 iPhone
AEI on Facebook
Events Calendar
  • 15
    MON
  • 16
    TUE
  • 17
    WED
  • 18
    THU
  • 19
    FRI
Tuesday, September 16, 2014 | 5:00 p.m. – 6:00 p.m.
The Constitution as political theory

Please join us for the third-annual Walter Berns Constitution Day Lecture as James Ceasar, Harry F. Byrd Professor of Politics at the University of Virginia, explores some of the Constitution’s most significant contributions to political theory, focusing on themes that have been largely unexamined in current scholarship.

Wednesday, September 17, 2014 | 8:10 a.m. – Thursday, September 18, 2014 | 1:30 p.m.
Third international conference on housing risk: New risk measures and their applications

We invite you to join us for this year’s international conference on housing risk — cosponsored by the Collateral Risk Network and AEI International Center on Housing Risk — which will focus on new mortgage and collateral risk measures and their applications.

Event Registration is Closed
Thursday, September 18, 2014 | 2:15 p.m. – 3:00 p.m.
Speaker of the House John Boehner on resetting America’s economic foundation

Please join us as Speaker John Boehner (R-OH) delivers his five-point policy vision to reset America’s economy.

Event Registration is Closed
Friday, September 19, 2014 | 9:15 a.m. – 11:00 a.m.
Reforming Medicare: What does the public think?

Please join us as a panel of distinguished experts explore the implications of the report and the consumer role in shaping the future of Medicare.

Event Registration is Closed
No events scheduled this day.
No events scheduled this day.
No events scheduled this day.
No events scheduled this day.