Data: No deus ex machina

Standardized test by

  • Title:

    Cage-Busting Leadership
  • Format:

    Paperback
  • Paperback Price:

    29.95
  • Paperback ISBN:

    978-1-61250-506-0
  • 256 Paperback pages
  • Buy the Book

Article Highlights

  • Data can be a powerful tool. But we must recognize that collecting data is not using data.

    Tweet This

  • It seems clear that would-be reformers have consistently overestimated the potential of data.

    Tweet This

  • How would a #cagebusting leader use data in school systems?

    Tweet This

  • Would-be reformers too often suggest that data is going to resolve long-standing problems.

    Tweet This

 

Data-based decision-making is all the rage. Secretary of Education Arne Duncan (2009) has emphatically declared, "I am a deep believer in the power of data to drive our decisions. Data gives us the roadmap to reform. It tells us where we are, where we need to go, and who is most at risk." In the past few years, all 50 states have adopted most or all of the Data Quality Campaign's framework for state data systems.

In important respects, this is a welcome development. Data expose inequities, create transparency, and help drive organizational improvement.

But something is amiss. Many educators regard talk of data-based decision-making as an external imposition, sensing new obligations and what they see as a push to narrow schooling to test scores and graduation rates. Districts remain hidebound and bureaucratic, with precious few looking like data-informed learning organizations. And the data—which are relatively crude, consisting mostly of reading and math scores—are unequal to the heavy weight they're asked to bear.

Despite these challenges, enthusiasts continue to make sweeping claims about the restorative power of data. Too often, as we talk to policymakers, system leaders, funders, advocates, and vendors, we get a whiff of deus ex machina, the theatrical trick of having a god drop from the heavens to miraculously save the day. (The phrase's literal meaning is "God in the machine.") Like a Euripides tragedy in which an unforeseen development bails out the playwright who has written himself into a corner, would-be reformers too often suggest that this wonderful thing called "data" is going to resolve stubborn, long-standing problems.

We need a more measured view. Data can be a powerful tool. But we must recognize that collecting data is not using data; that data are an input into judgment rather than a replacement for it; that data can inform but not resolve difficult questions of politics and values; and that we need better ways to measure what matters, rather than valuing those things we can measure.

We've Been Here Before

Data have long promised easy answers, sometimes with discomfiting results. Frederick Kelly created the first modern multiple-choice test in 1914 (Murdoch, 2007). Others quickly followed suit. Edward Thorndike and Charles Judd devised achievement tests in spelling, handwriting, arithmetic, composition, and more (Butts & Cremin, 1953). By 1923, more than 300 standardized scales were available (Cubberley, 1919).

Stanford's iconic dean of education, Ellwood Cubberley (1919), cheered such assessments, insisting, "We can now measure an unknown class and say, rather definitely, that, for example, the class not only spells poorly but is 12 percent below standard" (p. 694). Cubberley explained,

Standardized tests have meant nothing less than the ultimate changing of school administration from guesswork to scientific accuracy. The mere personal opinions of school board members and the lay public … have been in large part eliminated. (p. 698)

Consider the IQ test, created to help sort new recruits mobilized for World War I. The U.S. government asked elite psychology professors to develop a system for gauging intelligence. In hindsight, some of the results were unreliable. In one analysis, testing expert H. H. Goddard identified 83 percent of Jews, 80 percent of Hungarians, and 79 percent of Italians as "feeble-minded" (Mathews, 2006). In one 1921 study, Harvard researcher Robert Yerkes concluded that "37 percent of whites and 89 percent of negroes" could be classified as "morons" (Gould, 1981, p. 227). Yerkes had no concerns about the results because the tests were "constructed and administered" to address potential biases and were "definitely known to measure native intellectual ability" (Graham, 2005, p. 48).

In the 1960s and 1970s, proponents of data and accountability again insisted that they had it right. U.S. Office of Education Associate Commissioner Leon Lessinger (1970) promised,

Once we have standardized, reliable data on the cost of producing a variety of educational results … legislators and school officials will at last be able to draw up budgets based on facts instead of on vague assertions. Through knowledge gained in the process of management, we will also be able to hold the schools accountable for results. (p. 10)

Lessinger was hardly alone; more than 4,000 books and articles on data and education accountability were published in the late 1960s and early 1970s (Browder, 1975). Yet in 2001, No Child Left Behind's architects started from the bipartisan conviction that U.S. schooling was nearly bereft of good data.

In hindsight, it seems clear that would-be reformers have consistently overestimated the potential of data and have used new data in inappropriate and troubling ways. We'd do well to keep this in mind if we intend to do more than repeat past mistakes.

The full text of this article is available via subscription at Education Leadership.

Frederick M. Hess is director of education policy studies at the American Enterprise Institute and writes the Rick Hess Straight Up blog for Education Week. Jal Mehta is an assistant professor at the Harvard Graduate School of Education and coeditor, with Robert B. Schwartz and Frederick M. Hess, of "The Futures of School Reform" (Harvard Education Press, 2012).

Also Visit
AEIdeas Blog The American Magazine
About the Author

 

Frederick M.
Hess

What's new on AEI

Love people, not pleasure
image Oval Office lacks resolve on Ukraine
image Middle East Morass: A public opinion rundown of Iraq, Iran, and more
image Verizon's Inspire Her Mind ad and the facts they didn't tell you
AEI on Facebook
Events Calendar
  • 21
    MON
  • 22
    TUE
  • 23
    WED
  • 24
    THU
  • 25
    FRI
Monday, July 21, 2014 | 9:15 a.m. – 11:30 a.m.
Closing the gaps in health outcomes: Alternative paths forward

Please join us for a broader exploration of targeted interventions that provide real promise for reducing health disparities, limiting or delaying the onset of chronic health conditions, and improving the performance of the US health care system.

Monday, July 21, 2014 | 4:00 p.m. – 5:30 p.m.
Comprehending comprehensive universities

Join us for a panel discussion that seeks to comprehend the comprehensives and to determine the role these schools play in the nation’s college completion agenda.

Tuesday, July 22, 2014 | 8:50 a.m. – 12:00 p.m.
Who governs the Internet? A conversation on securing the multistakeholder process

Please join AEI’s Center for Internet, Communications, and Technology Policy for a conference to address key steps we can take, as members of the global community, to maintain a free Internet.

Event Registration is Closed
Thursday, July 24, 2014 | 9:00 a.m. – 10:00 a.m.
Expanding opportunity in America: A conversation with House Budget Committee Chairman Paul Ryan

Please join us as House Budget Committee Chairman Paul Ryan (R-WI) unveils a new set of policy reforms aimed at reducing poverty and increasing upward mobility throughout America.

Thursday, July 24, 2014 | 6:00 p.m. – 7:15 p.m.
Is it time to end the Export-Import Bank?

We welcome you to join us at AEI as POLITICO’s Ben White moderates a lively debate between Tim Carney, one of the bank’s fiercest critics, and Tony Fratto, one of the agency’s staunchest defenders.

No events scheduled this day.
No events scheduled today.
No events scheduled this day.
No events scheduled this day.
No events scheduled this day.