<html><body><P>American Enterprise Institute</P> <P>May 21, 2007</P> <P>[Edited transcript from audio tapes]</P> <P><BR> <TABLE cellSpacing=1 cellPadding=1 width="100%" border=0> <TBODY> <TR> <TD> <DIV class=BodyText>8:30 a.m. </DIV></TD> <TD>Registration</TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD>9:00</TD> <TD> <DIV class=BodyText><EM>Introduction</EM>: </DIV></TD> <TD> <DIV class=BodyText>Frederick M. Hess, AEI</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText>9:10</DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD><STRONG>Panel I: The Evolving Relationship between Research and Policy</STRONG></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText><EM>Presenters</EM>: </DIV></TD> <TD> <DIV class=BodyText>Jeffrey Henig, Columbia University Teachers College</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText>Andrew Rudalevige, Dickinson College </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText><EM>Discussants</EM>:  </DIV></TD> <TD> <DIV class=BodyText>Gina Burkhardt, Learning Point Associates </DIV> <DIV class=BodyText>Michael Feuer, National Research Council</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText>Ellen Condliffe Lagemann, Harvard University</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText>10:35  </DIV></TD> <TD> <DIV class=BodyText>Break</DIV></TD> <TD></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText>10:45</DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText><STRONG>Panel II: How Research Is Used Teacher Quality and Reading</STRONG></DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText><EM>Presenters</EM>:  </DIV></TD> <TD> <DIV class=BodyText>Richard Ingersoll, University of Pennsylvania</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText>James Kim, Harvard Graduate School of Education</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText><EM>Discussants</EM>:</DIV></TD> <TD> <DIV class=BodyText>Reid Lyon, Higher Ed Holdings and Whitney International University</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText>Lorraine McDonnell, University of California, Santa Barbara</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText>12:00 p.m. </DIV></TD> <TD> <DIV class=BodyText>Luncheon</DIV></TD> <TD></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText>12:45 p.m.</DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText><STRONG>Panel III: How Research Is Used NCLB and School Choice</STRONG></DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText><EM>Presenters</EM>: </DIV></TD> <TD> <DIV class=BodyText> <DIV class=BodyText>Michael J. Petrilli, Thomas B. Fordham Foundation</DIV></DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText>Andrew Rotherham, Education Sector</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText><EM>Discussants</EM>:  </DIV></TD> <TD> <DIV class=BodyText>David Driscoll, Massachusetts Commissioner of Education</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText>Roberto Rodriguez, United States Senate HELP Committee   </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText>2:00  </DIV></TD> <TD> <DIV class=BodyText> <DIV class=BodyText>Break</DIV></DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText>2:10</DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText><STRONG>Panel IV: How Research Is Used by the Public, the Courts, and Educational Leaders</STRONG></DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText><EM>Presenters</EM>: </DIV></TD> <TD> <DIV class=BodyText>Lance Fusarelli, North Carolina State University </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText>Joshua Dunn, University of Colorado at Colorado Springs</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> <DIV class=BodyText><EM>Discussants</EM>: </DIV></DIV></TD> <TD> <DIV class=BodyText> <DIV class=BodyText>Pascal Forgione, Austin Independent School District </DIV></DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText>William Howell, University of Chicago</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText>Warren Simmons, Annenberg Institute for School Reform</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText>3:30  </DIV></TD> <TD> <DIV class=BodyText> <DIV class=BodyText>Break</DIV></DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText>3:40</DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText><STRONG>Panel V: Changing the Incentives for Researchers and Decision-Makers</STRONG></DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText><EM>Presenters</EM>: </DIV></TD> <TD> <DIV class=BodyText>Dan Goldhaber, University of Washington</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText>Kenneth Wong, Brown University</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText><EM>Discussants</EM>: </DIV></TD> <TD> <DIV class=BodyText>Michael McPherson, Spencer Foundation</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText>Kathleen McCartney, Harvard Graduate School of Education</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText>Grover J. (Russ) Whitehurst, Institute of Education Sciences</DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText>5:10  </DIV></TD> <TD> <DIV class=BodyText> <DIV class=BodyText>Reception </DIV></DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR> <TR> <TD> <DIV class=BodyText>6:00  </DIV></TD> <TD> <DIV class=BodyText> <DIV class=BodyText>Adjournment</DIV></DIV></TD> <TD> <DIV class=BodyText> </DIV></TD></TR></TBODY></TABLE></P> <P> </P> <P>Proceedings:</P> <P>[Start of Panel 1:  The Evolving Relationship between Research & Policy]</P> <P>Frederick M. Hess:  I m Rick Hess, director of Education Policy Studies at the American Enterprise Institute.  I would like to welcome you all here today for this conference, The Politics of Knowledge:  Why Research Does (or Does Not) Influence Education Policy. Before we get started we are going to get started in about five or ten minutes I would like to take a few moments to explain why we have organized this conference and what I hope we might accomplish today.  </P> <P>Before we do that, I think it is important to suggest that when we deal with research or knowledge, we are always building upon prior efforts.  And for those of you who are confused, this is actually not simply a reenactment of Ellen s book The Politics of Knowledge.  We are simply appropriating a title because that is how we do things in the academy.  </P> <P>Now, what are we talking about today?  Well, we have spent a great deal of time in recent years discussing the merits of various research methodologies and ways to enhance the rigor and the reliability of research and evaluation.  We have also spent a good deal of time discussing how to communicate findings to practitioners, to schools and to classrooms so that new findings will actually be utilized.  We have spent far less time thinking seriously about our efforts to promote the rigor and relevance of research, our position within the political and policy environment, about how political pressures shape the research agenda, how research is communicated or fed into the policy-making process, or how relationships between researchers and public officials and advocates and educators and the media affect the utility and the use of research.  Instead, we have been more likely to hear or to voice broad-brushed enunciations of the educational research community, of think tanks, or of politicized decision-making.  </P> <P>Our focus today is on the nexus of research in policy-making, on the soft tissue that links these two worlds, at the institutions, norms, and incentives that shape the respective activity.  The guiding assumption is that we can get all the technical and scientific questions right and still not benefit from scholarly activity and conversely, that the value of research will be dramatically heightened if the incentive, the norms, and the institutions are sensibly constructed.  </P> <P>I do not anticipate that anybody is going to leave today with simple answers.  I do hope that we leave better-equipped to address the question: If we really believe that valid and reliable research findings can and should influence education policy in various ways, what arrangements or norms will help to make it so?  </P> <P>Similarly, I hope we might also be prepared to say that, if we believe that research should be fair-minded, carefully constructed, and skillfully executed and we are willing to go to where the data points us, and able to stretch our understanding to what we ought to be doing and how we ought to do it, what arrangements or norms will encourage the production of such work?  If we leave today with useful insights into these two questions, I will consider the day a success.  </P> <P>With that, let me quickly give you the rundown on how the day is going to go.  What we are going to have is five panels, starting with the panel you see seated here.  </P> <P>The first panel We are going to look at the evolving relationship between research and policy and talk a bit about how we have gotten to where we are today.  The second panel we are going to talk about how research is used or has been used in the case of teacher quality and reading.  The third panel we are going to discuss how research has been used in the case of NCLB and school choice.  Panel four We are going to think about how research has been used by the public, the courts, and educational leaders, and finally, we are going to see if we can get some larger take-aways for the day.  We are going to think about how we can change the incentives that face researchers and policy-makers.  </P> <P>Each panel is going to feature a couple of authors and the discussants, all of them speaking for about 10 minutes a piece.  We then are going to open it up for conversation among the panelists and for Q&A.  During Q&A, I would ask that you please wait for the microphone.  We will have two mics circulating.  And then do us the favor of identifying yourself by name and affiliation.  I ll also ask because I always do that we please avoid the proud DC tradition of asking a question by making a speech from the floor and then appending a question mark and I ll ask that we will actually ask questions which can promote and advance the conversation.  It is understood that in a day like this that there are a number of people in the room that could quite readily sit up here.  I see Jim Kohlmoos out there.  I see a variety of folks who thought deeply about these issues.  I find these days tend to be much more productive, however, if we avoid the temptation to orate from the floor.  </P> <P>The papers that are being presented will be revised and then published as a collective volume in early 2008.  Because the drafts that you see presented here today are still very much works in progress, I would invite audience members during the day to please feel free to corner authors and suggest corrections, offer thoughts and please challenge analyses and interpretations.  The entire set of papers is available online at the web address <A target=_blank href="http://www.aei.org/event1455" target=_blank>www.aei.org/event1455</A>.  That is event 1455.  It is also really easy to find if you go to the AEI website.  For those of you who prefer not to deal with the web, we have got a full set of papers and hard copies outside where you came in the foyer and we have also got CDs with the full set of papers available for those who prefer something small and in their hand.  </P> <P>Finally, before I turn to the first panel, I would like to thank The Spencer Foundation for helping to make this conference possible and I would like to thank the generous donors who support AEI s education program more generally.  I would also like to thank Juliet Squire standing up here with the mike for her outstanding efforts in quarterbacking this whole thing and organizing it and in getting all these people safely and in orderly fashion in DC, so thanks, Julie.  </P> <P>If you have questions during the day, Julie, hopefully can be able to help you out.  Also, Morgan Goatley right here, you can corner.  And Rosemary, where are you?  There you are.  Rosemary.  And all of them are far more competent than I so if you have any questions or concerns, hopefully they can help you address them.  </P> <P>With that, we are going to roll right into the first panel.  Let me briefly make the introductions.  .  </P> <P>Andy Rudalevige is an associate professor of political science at Dickinson College where he has taught since 2000.  In 2004-05, he was a visiting research scholar at Princeton University s Center for the Study of Democratic Politics.  His first book Managing the President s Program published by Princeton University Press won the American Political Science Associations Richard E. Neustadt Award.  </P> <P>Jeff Henig - seated to Andy s right - is a professor of political science and education at Teachers College at Columbia University and a professor of political science at Columbia University.  He is the author or co-author of seven books including the Color of School Reform: Race Politics and the Challenge of Urban Education and Mayors in the Middle:  Race Politics in Urban School Reform, both published by Princeton University Press.  </P> <P>Our discussants for the first panel Speaking first will be Ellen Lagemann.  Ellen is a Charles Warren professor of the History of American Education at Harvard University.  In July, she will become a distinguished fellow at the Bard Center where she will direct the Bard Center for Education and Democracy.  A historian of education, Ellen is a former dean of the Harvard Graduate School of Education and former president of The Spencer Foundation. </P> <P>Speaking second will be Gina Burkhardt.  Gina is chief executive officer of Learning Point Associates.  Prior to becoming CEO of Learning Point in 2003, Gina served as executive director of the North Central Region Educational Laboratory at Cornell.  </P> <P>And speaking last on this initial panel will be Michael Feuer.  Mike is executive director of the Division of Behavioral and Social Sciences and Education in the National Research Council of the National Academies.  Formerly at the National Research Council, Mike was the first director of the Center for Education and the founding director of the Board on Testing and Assessment. </P> <P>With that, Andy, will you please get started?  </P> <P>Andrew Rudalevige:  Good morning!  My name is Andy Rudalevige.  It is a great pleasure to be here this morning.  My assignment here today was to talk about historical and political topography of education research as it has been carried out by the federal government, most specifically, within the Department of Education and its various organizational ramifications.  I will not talk so much about other places in the government where education research is carried out NSF, NIH, especially, but we will talk about those touchstones for the way in which education research has been conceived by successive generations of reformers.  </P> <P>As you can see the paper is grandiosely titled Truth versus Partisanship, and you will see where that comes from in a minute.  For a Washington audience, I do not think I need to tell the story of the three envelopes.  Anyway, given our Draconian or I should say Hessian limits on time, I think I will have to skip over it but most of you know the punch line, which is that when someone brought into office needs to get advice from his or her predecessor, the predecessor leaves some envelopes on the desk.  The first one says, Blame your predecessor.   The second one says, Reorganize.  Reorganize.  Reorganize.   And the third one says, Prepare three envelopes.   </P> <P>The three envelopes have been in constant use in the federal education research bureaucracy forever, really, but especially since 1965. I have considered the sequence of organizational manifestations of this function from the Bureau of Research consolidated in the Office of Education back in the mid-60s, the creation of an independent National Institute of Education back in 1972 that was then moved under a new office the Office of Educational Research and Improvement, OERI, in 1980, when the new Department of Education came into being.  Five years later NIE was wiped out off the organizational chart OERI sorry filled that void itself.  A major reorganization less than ten years later tried to model OERI along lines of the National Institutes of Health with specific subject matter institutes under its umbrella.  And then less than a decade later, the creation of the Institute of Education Sciences, IES, its current manifestation.  </P> <P>Now, all those changes do have an underlying theme and that is the quest for science.  This has been something as a grail for politicians especially.  You can see there recent quotes just from the 2001 2002 debate over the education sciences reformat, the very name of which gives you an idea of what political actors were trying to accomplish in this latest reorganization.  Again, the idea is to insulate federal research from partisan or political influences.  To look into this institute when we have education questions the same way that we look to the NIH when we have medical questions.  In this striving for a scientific agency which will be free of partisanship and produce real facts has been a hallmark of this debate, not just in the 2002 debate but going back well into American history.  </P> <P>The title of the paper comes actually from a section of a report, an advisory committee to President Hoover, back in the early 1930s.  Section entitled (Truth versus Partisanship), so you can see the idea of the committee strongly influenced by notions of scientific management and the idea that there was one right way to do pretty much anything argued that we need to get beyond mere differences of opinion tenaciously held and we can do this how?  Well, by finding out facts, facts that are established by the scientific method and presented in understandable terms.  Clearly, something that would fit in very nicely with the idea of evidence-based interventions in the current round of educational accountability in research.  This was to be done organizationally as well.  The research function needed more clout.  Right, it needed enough status that they could present those facts forcibly both to the Congress and to the president.  </P> <P>Now, in point of fact, educational research has had a hard time using force in quite that way.  One question of the paper and I will not go into a huge amount of detail here is the question of whether structure can in fact serve science in this context.  Terry Moe, a Stanford political scientist, wrote some years ago about the politics of bureaucratic structure and the difficulty in achieving insulation and autonomy from political forces when agencies are created or reorganized, right?  Because structure matters to outcomes, we should expect that structure itself is a variable that is contested.  That political actors of all stripes, widely construed, ought to be interested in what an agency looks like and in fact, you have seen that striving over time.  </P> <P>In education research, rarely have you seen the insulation that you have seen in the hard sciences so I ll come back to that.  Partly, of course, as we ll talk about more, education research is hard to pull off in a scientific manner.  Right, it is hard to do controlled experimental design in a lot of cases and besides that it is value-laden.  Right?  So a lot of the research agendas are driven by values and things that move past the normal realm of determining facts and that makes it inherently political and that is also meant to be politicized in a lot of ways.</P> <P>Going back to the earliest incarnation of the Department of Education, it was actually originally a department at least in name As early as 1867, it was supposed to collect statistics and facts but in fact, in time, it collected all sorts of weird extraneous functions including the reindeer service in the Alaskan territory, which dealt with the purchase and the breeding of reindeer for native populations in Alaska.  They did not get rid of this until the end of the 20th century.</P> <P>In the modern era, the Office of Education, NIE, OERI, there has been something of a vicious cycle where incentives to politicize have been more common than incentives to insulate.  There has not been necessarily a lot of organized interest support except with the regional labs in the R&D centers which we can talk more about.  Legislative interest has been, perhaps for that reason, somewhat sporadic, except for the occasional opportunities to look for ear-marking and chances to benefit one s district.  </P> <P>There has been lots of turnover within the personnel structure of the Federal Research Establishment and partly as a cause, and partly as a consequence of that perhaps, research really has been seen as a political arm of the administration and as unscientific any way.  Carl Kaestle of course wrote about the awful reputation of education research; that certainly has not helped the function gain support legislatively or even administratively in a lot of cases. </P> <P>What we have seen really is institute-envy.  We have had a repeated desire for NIH and the debate is going back to the Office of Education, NIE, OERI, and now IES, again you see the same arguments made  Why can this not be more like NIH?  Sometimes NSF but NIH more regularly, given its multiple umbrella organizations.  Can we insulate it from partisan-buffering?  Can we have appointments that are not political?  Can we have fixed terms for example?  Keep them out of the normal political crush?  Can we have some advisory board that is prestigious and serves to buffer the organization from politics?  We need some rigorous peer review.  How can we make that happen?  </P> <P>And we need lots of time and patience, and frankly, money. I m from Massachusetts, so I can say this, we have got something like New Hampshire NH, not NIH Live free or die.  And in fact that has done both, right?  Lived, and lived practically without money and as a consequence, or perhaps wither on the vine.  I can say other nasty things about New Hampshire later but that will do for now.</P> <P>How might we lead to the politics of insulation?  Well, the last round in the early 2000s did seem to lead despite this checkered history of a condition where there have been efforts to move in steps down this road.  Partly perhaps, this is caused by desperation the idea that we are running out of ways to this, ways to think about this.  Three strikes and perhaps we are out.  Our next effort might be the last one where the federal government seriously pays attention to this function even though it is perhaps the one function that everyone agrees ought to be federalized.  </P> <P>There were at that time, of course, some high profile dividends that seemed to be flowing from education research, especially in the reading sphere.  Most of them not funded by the way by the Department of Education itself.  You had of course, an on-going, very salient debate about measurements and standards and the idea that you needed some scientific underpinning to that measurement and to those standards, which coincided of course, with the political window.  Before President Bush s election and Congress preceding him, there was relatively bipartisan agreement over the things that would later go into No Child Left Behind. But add to that mix President Bush and his real focus on measurement especially and on accountability, and create a period where you could have something occur like No Child Left Behind, which stresses multiple times its goal of getting scientifically based research into the mix.  </P> <P>So the ESRA, the Education Sciences Reform Act in a lot of ways is unique in this sequence or cycle as a bipartisan vehicle where members of Congress actually wanted to claim credit for doing something along these lines.  In the past, most members of Congress did not care very much.  NIE was achieved largely by the ideological entrepreneurship perhaps of John Brademas, a Congressman from Indiana.  There were more people interested this time around in part because it did provide opportunity and a vehicle for members of Congress to say, Yes, we are doing something tangible towards accountability, towards all the nice rhetoric that we have talked about in No Child Left Behind. So, within IES and the watchful gaze of Russ Whitehurst, we have science for the first time really in statute and a lot of definitions of what that might entail, again, with an emphasis on evidence-based interventions and the notion of controlled experimentation.  </P> <P>A fixed term for a presidentially-appointed director and the NCES commissioner, also has a fixed presidential term that is staggered away from presidential elections.  The director has moved up to level 2 of the executive schedule, right up from level 4 and prior to the last organization before that, up from level 5.  Again, same as the NSF, an advisory board with a majority of researchers created.  This is actually new in this history and the mission of the agency streamlines somewhat to remove technical assistance, the ORAD function [sounds like] and adding evaluation more explicitly so that the idea that IES could in fact do this function in an isolated way, at least had a chance to take off.  </P> <P>So far, there is obviously some criticisms that I will come to in a minute So far, so good seems to be at least the broad consensus of the education research community.  Do I wait for corrections on that front?  Whitehurst himself called this a tipping point.  That enough things had come together that we could finally move towards serious education research at the federal level.  There has been some independents [sounds like] of the agency vis-�-vis what works -- the whole reading recovery, controversy was resolved by the What Works Clearinghouse at least in a way of saying.  </P> <P>Well, it did work despite the political tides against that finding.  Peer review has been revamped.  Implementation as I mentioned has been separated from evaluation, all things that least make the creation of science possible.  Others would argue perhaps that Well, it is not a tipping point.  The whole thing is tipped over.   There is too much science and what there is, is too narrowly defined, that education does not work this way, that there is too much centralized control, whether it is over the education statistics function over the labs and the centers, whose missions have been redefined centrally.  It is argued that the independence might lead to too little departmental support.  </P> <P>Moving forward and you see a little bit of that perhaps in the departmental strategic plans that are moving forward and the idea that this might lead to administrative micromanaging.  You could also argue that the centralized control issue will lead to more legislative micromanaging as the labs and centers perhaps make their stand.  </P> <P>So key questions looking forward and I will stop here Will incentives continue to match the needs of the research community?  That is going to be the important thing moving forward and at this point, the jury is perhaps out.  And finally, politics have been amenable to the structure we have in place.  Will that continue to be true?   And even if the structure is a good one, will it have the resources that it needs to succeed?  And we end here with the question that eliminates any question of politics whether it is the politics of knowledge or whatever else, the politics of defining politics itself, who gets what?  When?  Where?  How?  Will education research get its, share?  That again will be a huge question moving forward.  Thank you very much.  </P> <P>Frederick M. Hess:  Thank you Andy.  Jeff?  And again, while we get Jeff set up, let me mention that I do impose Draconian time limits on everybody.  It is just one of those things.  Let s see.  I have got to be Draconian on some sense or other and but you will find copies of all the papers available out on the foyer.  You will find again, as I said, CDs with the full set of papers if you are interested.  And as we mentioned, all the papers are available online at the AEI website.  Jeff.</P> <P>Jeffrey Henig:  Okay.  So when it comes to the role of research and shaping public policy and debate, one might reasonably argue that this is the best of times.  NCLB with its frequent mention of evidence-based decision-making has authoritatively underscored the important role that objective knowledge could and should play in a democratic society.  As Andy suggested, IES with its grand policies and promotion of randomized field trials with the What Works Clearinghouse has provided a detailed road map of what strong research design might look like in education policy.  </P> <P>Research findings and research debates get deep coverage in outlets like Education Week and instant coverage in the blogosphere.  Some researchers are sufficiently visible in fact to have become mini-celebrities in their own right with their pictures on the front page of the Wall Street Journal.  Or it might just as well be argued that this could be the worst of times.  While highly visible, research, as it appears in the public stage, often presents an unpretty face.  </P> <P>I lead my paper with four mini-dramas little research battles that have played out in the front pages of real life education policy research.  Paul Peterson versus John Witte on the front page of the Wall Street Journal, combating one another in terms of research on vouchers in Milwaukee, with Peterson calling Witte a snake and Peterson saying Witte s research is lousy.  Second mini-drama, the AFT charter school study on the front-page of the New York Times.  A week later, Center for Education Reform taking out a full page ad in New York Times decrying both the research and Times coverage.  Front page of the Wall Street Journal again, Jesse Rothstein, Princeton economist, versus Caroline Hoxby, Harvard economist, combating one another on Hoxby s earlier research on intrametropolitan school competition, again, with charges of name-calling and ideological bias.  </P> <P>And the fourth mini-drama I start with relates to the Reading Wars and Reading First, which I label The US Department of Education versus itself.  I will not go to the heart-pumping details of these now.  I will leave it to say that each tells a story of prominent attention to the importance of high-quality research with a flipside portrayal of the research enterprise as personalized, partisan, and bombastic.  In a book that should be out early next year, I look at this in detail with particular attention to the Charter School debate and the polarized presentation of research around school choice and charter schools.  </P> <P>In this paper, I try to distill some lessons from that analysis to highlight six broad structural changes that I think are potentially changing the demand for research, the availability and type of data, and the way research enters the public realm as part of on-going policy and political debates.  </P> <P>Briefly, these are new technologies and the dissemination of research.  Think tanks, advocacy organizations and others have electronic newsletters.  You know all of these: PEN s weekly Newsblast which has an estimated 240,000 readers Checker Finn s Education Gadfly and Andy Rotherham s blog, Eduwonk, which gets about 1,200-1,400 visitors everyday.  One indicator of the scope of the emerging virtual world of education policy communication:  Eduwonk includes 97 links to other education blogs, 12 links to sites providing education news and analysis and 30 policy and political blogs that cover education along with other issues.  </P> <P>I speculate in the paper that this may have altered the relationship between research and policy in at least two ways.  One is speed.  There is a palpable sense of time pressure felt by some researchers that once one study is out and disseminated quickly, that in order to quickly provide an antidote, it is necessary for them to get their evidence out just as quickly.  I think there is an argument also that the speed contributes to an erosion of quality-control.  </P> <P>In general, while these avenues of getting research directly from researchers to the public, bypassing journals, bypassing peer review, things like that, represents in some ways a refreshing democratization of the debate about research.  It matters at a point where some studies are circulated and cited in early drafts, not only before peer review, but before even careful checks of the data and analysis.  </P> <P>The second broad structural change I talk about in the paper is the academy as a context and I discuss such changes as a shifting relationship between disciplinary and interdisciplinary programs, declining status of ed schools.  What I call hyperpluralism, multiple and competing journals and subfields and speculate in the paper about how these may contribute to a dysfunctional fragmentation of the landscape of academic and professional life and with consequences, I think, both for how the research unfolds within the scholarly community and more immediately to the point here, undermining the image of research, particularly academic research and its utility to journalists, policy-makers and citizens.  Academic research on education policy is quite literally all over the place.  </P> <P>Third broad structural factor I talk about in the paper has to do with privatization and the growth of the corporate sector in K-12 education.  You all know the story:  increasing role of for profit education management organizations, publishing industry, for profit providers of professional development, curriculum testing, the whole new supplemental education services industry and direct delivery by for-profit and nonprofits through contracts and charter schools, and the like.  </P> <P>I suggest that on the positive side these provide a new market for education research.  If you are a young scholar coming out with the proper skills, there are a lot more places for you to look for jobs and for people who are interested in what you are doing than in the past.  But I also suggest that there is a risk that this puts new constraints on research and researchers.  For example, in terms of access to what is often regarded as proprietary data.  </P> <P>Fourth broad structural factor I talk about is where Andy left off in terms of federal funding, I think, again, the emphasis on evidence based on stronger research designs is leading to an affirmation of the importance of research but government funding is tight and constrained.  For every hundred dollars in federal money going to research less than 41 cents goes to research in the Department of Ed.  There is little of that money very little of that money as I talked about in the paper that goes to basic research, more open-ended theoretically-driven research, whereas for the applied and developmental end of the research a large percentage, more than half in dollars, is going out in the form of contracts rather than grants, with tighter oversight as a result of the research design, measurement, analysis, and the dissemination of the results.  </P> <P>Against this backdrop, at least in some high profile areas, researchers turn more to foundations for support of their research but that money is tight, too.  Many foundations prefer to put their funding into direct services.  Among those that are willing to fund research, there is increased emphasis within the research, within the foundation community, on the importance of making sure that what they do supports the foundation mission as distinct from, in some instances, from the attitude of some of the early foundations in terms of supporting social learning.  Increasingly the foundations -- because they are interested like anyone else is in bang for the buck, put their attention, even their research attention, into dissemination and advocacy.  </P> <P>The fifth broad factor and I know there is a panel later on the courts I will be even more cursory here than I have been on the others is the courts as a customer.  Researchers do respond to demand from government for different information and during the 60s, a lot of the attention from the courts was looking for data relevant to the cases on school desegregation.  I argue in the paper that arguably there is a shift in the action from the courts, from the feds, to the states a shift in focus from race to finance and a shift from equity to adequacy to the extent that these put the focus on cost-benefit ratios and economic models.  </P> <P>These may actually be lowering the temperature to some extent on research compared to the 60s and 70s when research was entangled in the hot issues of race and busing, but to the extent that they develop as they may into broad consideration of such issues as to what does it take to be a full citizen or to the extent that they lead to a strong judicial mandates for higher funding and subsequently higher taxes, these shifts have the potential to embroil research in a new set of controversies.  </P> <P>The last structural change I talked about has to do with the dynamics of federalism, discussion about education, and education research is moving up the ladder of federalism from the locals to the states to the nationals.  I argue in the paper that at the local level, research is often largely constrained.  The local education bureaucracies have a tight control on data and as result, researchers find it often a thin environment unless they align their research with specific applied interest of the local district.  The local arena of discussion is often somewhat of a more pragmatic one than one that we see on the national level where the debate is highly visible, highly partisan.  Research is often seen as a weapon rather than as a tool for answering straight-forward questions about what works or what does not.  </P> <P>So, Rick encourages us to have a bang-up conclusion.  This is the best I could do.  </P> <P>Frederick M. Hess:  That is our boiler plate slide.  That is correct.  </P> <P>Jeffrey Henig:  So, is it the best of times or is it the worst of times?  My view is that the currently mixed picture is partly a product of two good things.  Researchers have heard the message that they should descend from the Ivory Tower and engage the world.  The old model of speaking truth to power where a scholar whispers in the ears of the power that be, the elite leaders, I think is also past.  In the age of the internet, discourse about research is democratized, is more open, and for the most part, that is to the good.  But it is a volatile time when promising opportunities are twinned with definite dangers.  </P> <P>Many of the very aspects of Ivory Tower research that are so frustrating to many, such as the abstract concern for theory, the deliberately unhurried pace, fascination with technical aspects of research design, reliance on an internal network of peer review that can be stuffy, conservative, and reactionary intellectually reactionary at times and on journals where scholars pluck to one another in terms no one else understands, many of these characteristics that are so frustrating also play a role in maintaining a socially-valuable set of distinctions --  distinctions between research and advocacy, between the pursuit of knowledge and the pursuit of advantage, between sounding good and being right, and it is an open question to conclude in my mind, How far down the path of relevance researchers can travel without putting something of value at risk?   Thank you.</P> <P>Frederick M. Hess:  Thank you Jeff.  Ellen.</P> <P>Ellen Lagemann:  Sorry.  Andy, thanks to Rick.  He may be a tough task minister but it is not easy to fill a room like this before 10 o clock in the morning in Washington DC.  </P> <P>As Rick has mentioned, I wrote a book almost 20 years ago called The Politics of Knowledge.  It was a history of the Carnegie Corporation of New York, which was chartered as Dan Fallon knows to promote the advancement and diffusion of knowledge.  In working on this book, I quickly realized and I probably should have known it before I wrote this book, but I do not think I thought about it, that the corporation was not simply a venture fund backed by philanthropic money.  Its founders, trustees, and staff, on the one hand, and its successful and unsuccessful grant seekers on the other all had agendas and those agendas were not necessarily the same thing.  They sometimes conflicted and as a result, it was very clear to me very quickly that the Carnegie Corporation, like any funding agency, was engaged in a politics of knowledge.  </P> <P>And I would like to build my comments about Jeff and Andy s papers around the conception of the politics of knowledge that I developed in that now very out-of-print book and it should stay out of print.  At least as I formulated in that book, the politics of knowledge is comprised of three interrelated sets of questions.  </P> <P>First, which fields of knowledge and approaches within different fields are more or less authoritative at different times and as a consequence, more or less relevant to policy-making?  Second, to what degree are our experts in different fields regarded and treated differently from non-experts?  That is a hugely important question in a democratic society.  And third, who can become an expert and how does that happen?  </P> <P>Viewed from the perspective of the politics of knowledge, education I believe and I m sad to say this but it is clear to me that education has come out on the short end of things.  I wish that were not the case for obvious reasons, I think but I think the evidence is very, very clear and some of this has been said.  Owing to its association with a feminized profession teaching and to its frequent association with schools that tend to be marginalized at the periphery of research universities, education research suffers from three major problems.  </P> <P>It has not drawn sufficient resources and by resources, I do not just mean money.  It has not gained high levels of respect, even when respect is deserved.  And it has too often failed to display the rigor needed to unravel the complicated problems that education presents.  </P> <P>Education research is, of course, an activity as several people have said that takes place in many different places, not just in schools and departments of education but all over universities, all over government agencies, all across think tanks, all across foundations.  People from many different disciplines, as well as education itself, also work in this field and as a result of all of this, I think education research has never developed a coherent set of standards or a clear consensus concerning central questions.  One must even wonder if it is accurate to speak of a distinct field called education research.</P> <P>The intriguing question posed by this conference and by Jeff and Andy s papers then is Are things getting better?   It has often been said that educational innovations tend to be polished by hope but unfettered by data.  Is work currently going on in the field - if it is a field - likely to change that sorry situation?  As I read their papers, both Jeff and Andy are guardedly optimistic.</P> <P>Jeff, who I read as being the more guarded of the two, believes the future caliber of research and its potential relevance and value in policy-making will depend on the balance achieved among opposing forces in what he calls in terms of six broad institutional arenas, which he just talked about.  </P> <P>Andy seems encouraged by changes in the institutional structures for education research, especially the autonomy purposely built into IES at its creation.  He rightly notes however that politics can undermine structure and the jury is still out on IES, which is still we must remember, particularly from a historical perspective, a very, very young agency.  It is hardly a nursery school.  </P> <P>I agree with most of what both Jeff and Andy have argued but I would also give more emphasis to two issues that have been raised but I do not think raised centrally enough, which in my view will have a hugely important impact on the future well-being of education research.  In my view, scholarship in education will or will not thrive depending on whether we can change what Carl Kaestle classically has called the awful reputation of education research.  To some extent, that will depend on the working out of the forces that Jeff and Andy have identified but it will also depend on larger matters of public perception.  So long as teachers are bashed and education is seen as a common-sense practice that pretty much anyone can do well, research about education will not be understood as the complicated, difficult, and very important undertaking that it is.  </P> <P>How then do we change centuries-old social perceptions that lead to the demeaning of expertise in education?  Partly, I think by doing excellent work and by being willing to criticize work that is not excellent and partly too, by talking as widely as we can about education research its importance, its demands, and its complexity.  Doing so can be extremely frustrating, as many people in this room know.  </P> <P>I remember talking at a Congressional briefing some years ago where a leading member in the House walked out after I challenged his view that all we needed to improve education was better teachers.  He stood up quite literally and said, Professor, why do you have to pretend it is all so complicated?   And he turned on his heels and he left.  Even when I did not win that mind or that heart that day, I probably would never win it I think speaking truth to power, not in the whispering in the ear sense but in the publicly-declaring sense, about education research is vital.  </P> <P>In addition, I think we need to admit that the canons of research that are likely to renew tenure or other credits in the academic and scholarly worlds are not the same as those that are likely to make your work valuable to policy-makers or to practitioners.  To actually be useable and maybe even useful in education, research must first be scientifically valid and reliable and then it must be translated to findings and recommendations that policy-makers can actually use or test toys or other things that practitioners can employ in their work.  </P> <P>In my view, gaining a better understanding of what usability means in education research and then finding ways to support such work is critical to the future of this field.  Useable knowledge generated by research is not likely to be tenure-able research so there are many disincentives to doing such work and overcoming those disincentives will require fundamental reform of universities.  Hardly a minor project if you have ever had a position of responsibility in a university.  </P> <P>But let me return to the two papers.  Let us hope the optimism, even the guarded optimism, in both papers is correct.  If it is, the likelihood of informing education policy-making with tested, disciplined knowledge that is based on solid, accumulating evidence will increase.  That said, as we all know or we would not be here, politics tends to make it very, very difficult to mount programs of high quality, reliable research in education that are useful and doing so will require sustained and determined effort.  It will also require what one of my favorite people, John Gardner, used to call loving critics who are willing to make tough judgments about the quality and the significance of the research that all of us do.  I hope that these papers and this conference can advance that project.  </P> <P>Frederick M. Hess:  Thank you, Ellen.  Gina.</P> <P>Gina Burkhardt:  So I took this task pretty seriously and came at it from a very concrete perspective so bear with me as I came up with a lot more questions than answers and I m going to put them out to the group and hopefully there will be some conversation. </P> <P>So I want to make four points.  I know you are only supposed to make three, but I could not get it down to three.  One is about politics.  One is about relationships.  One is about evolution.  And the last is about innovation.  </P> <P>So when I read these papers, I thought what came out of them was more an indication that to move education research forward and have a relationship with policy, you really need more of a knowledge of politics than a politics of knowledge -- and it goes back exactly to what Ellen had talked about earlier that there are these three arenas where knowing politics helps advance your cause, especially when you are inside baseball, which in my mind is inside the research community.  So if you figure out how to be the best researcher on the block or the researcher of the month, then it is your knowledge or your information that is highlighted and then often gets the most attention, whether or not it is targeted at policy-making or policy makers.  </P> <P>How you advance or move your research to the field is also politically, I think, dictated, so how it gets out mostly in scholarly journals up until this point where it is presented, how it is actually turned into application or utilization, which happens far less frequently than it should.</P> <P>And the third one is about who is involved or who is let into this elite community of researchers and that as we watch the field right now, we are watching different organizations come into or penetrate this elite field in terms of producing research and understanding the political nature of getting funded and getting charged with doing research.  </P> <P>The second point I would like to make is about relationships and as I look at relationships, I always think that it is best to have a relationship is with another person.  Having a relationship with yourself is good.  There are lots of self-help books out there that talk about loving yourself but I think the research community has gone a little bit overboard in terms of having a relationship with itself and less of a relationship with the outside world or the policymakers or practitioners that are highly dependent on the information to reform education as we know it.  </P> <P>Also, I think that the research community has figured out how to have a strong relationship with funders or with the Department of Education in terms of soliciting funding but they have not figured out very well respectfully, I say this how to disseminate or get the information out into the public so that it is used and applied.  </P> <P>And I would like to take issue a little bit with what was in the papers around regional education laboratories in the centers.  There has been this perpetuating myth that the regional education laboratories really were in a place of generating research over the last twenty years.  But up until this point, up until the new mandates from IES, they were really a set of research and development organizations and they were the one arm of the Department of Education that was really charged with doing the D in R&D and in our minds and I can say ours because I think my adult life has been spent in these communities in our minds, our job was to take what the research community put forward and actually work with policymakers at the state and local levels to create opportunities, to practice how the research was implemented, and to pilot how the research was implemented, so that we could watch policies in action.  What we did not succeed at and I put myself and our organizations among the many of you in the room, what we did not succeed at is the scale up and the cost of that, I think, has been significant to the education field.  </P> <P>The third point that I want to make is about evolution.  Organisms mostly, in a changing environment, adapt and change and again, I think the structures that both papers talked about are structures that did not adapt to the changing outside environments.  So we see a lot of shuffling inside the Department of Education around how best to structure the relationship between the department and the world and the research community.  </P> <P>And I think that the evolution process did not happen very well until very recently with the advent of IES and as part of again, a regional organization laboratory, I commend Russ Whitehurst and the group in IES for changing the conversation about the rigor of research and promoting research that is actually need-based.  If you look at the research being done now out of these centers and these labs, you will see that they are all mandated to be in response to needs that have been generated by the states and the locals.  </P> <P>What I think IES is still struggling with is the relationship part and their idea of a relationship still sits inside the research community.  They are working on, and we are helping them figure out, how best to have a relationship with policymakers.  That is above and beyond the political community because if you look at states, you will see that most of the policy-making around changes in reform and around compliance to NCLB happens at the state level, probably independent of a lot of the research that is happening in the research community or at the national level.  </P> <P>And my last point, which is really optimistic I think, is that I believe that innovation is going to trump all of these conversations and I m really looking forward to that happening.  We have technology that is advancing far more quickly than any of us can imagine and it is a process that is getting information out and the demand for information is growing.  There is a community sitting outside of this conversation businesses, entertainment, media that are really on the verge of converging and forcing really forcing change from the outside in.  </P> <P>So as we have these conversations sitting inside baseball, there is a larger force outside that is demanding with a lot of money that different kinds of education and different kinds of systems develop in response to the needs that all our kids have.  So as we move forward and we struggle with the idea of scientifically-based research and whether or not methodologies are the right methodologies, there is a lot of money, which is going into making change and reform happen in schools that is not necessarily based on research.  That is not a good thing.  So the communication between these outside forces and these inside communities really need to happen so that as we move forward, we are working together to make the changes work.  </P> <P>Frederick M. Hess:  Thank you, Gina.  Mike.</P> <P>Michael Feuer:  Let me add my thanks to Rick and the organizers for a very interesting conference and a very interesting set of papers.  My compliments to Jeff and Andy for two very excellent papers.  I m not going to try a page-by-page or a line-by-line commentary on these papers.  I might do that offline but I learned a lot from these papers.  I think they make very important contributions to the debate and to our continuing attempt to accumulate knowledge.</P> <P>In my 10 minutes this morning, I would also like to make four points.  That means about three minutes per point. </P> <P>Frederick M. Hess:  Did you try that line last night?</P> <P>Michael Feuer:  Like most of the work that I do I have already applied for a no-cost extension here so Point number one is about the several references to Carl Kaestle s wonderful 1993 paper called The Awful Reputation of Education Research.   There are more references to it actually in the written papers than you heard this morning but I think there is just the possibility a little bit of unintended sleight of hand going on with respect to the awful reputation of education research.  </P> <P>It has been a while since I read Carl s paper but as I remember it, Carl was quite distinctly making a difference trying to make a point that there is a difference between the reputation of education research and the quality of education research and in a lot of the rhetoric surrounding the debate that we are participating in today, one gets a sense that either in Carl s paper or in the way that it has been interpreted, that that reputation is deserved.  And I do not think actually that it is quite a correct reading of the original paper and it leads me to suggest two cautions about this business about the awful reputation. </P> <P>First of all, people like us who are writing about the quality of education research need to be just a bit more meticulous in the way that we understand the possibility for language in being interpreted and misinterpreted.  I think that constitutes -- an example of good quality research is when one pays more attention to the research being criticized.  Maybe more important, this distinction between research quality and the reputation of research is, I think, a key to understanding some of the weak points in the education research system.  From the standpoint of what I take to be an underlying assumption in this conference, which is that good research can and should make a big difference to policy, we would probably be better off studying the opportunities and constraints that affect the way research is used in our democratic system rather than try to come up with blanket indictments of the entire education research enterprise.  More simply put, even if we all agreed that some education research met the highest possible standards of quality, I would argue that that is at best a necessary but by no means a sufficient condition for assuring that that research would be understood, utilized, and applied in the real world.  </P> <P>Which brings me to my second point, much inspired by Jeff s mini-dramas which I really enjoyed a lot.  I think this is a beautiful reminder of the world we live in and by the way, Jeff, this one, I could not help point out that in your paper you talked about - in the Rothstein-Hoxby mini-drama something about the principal opponents and I think you are on to a new branch of statistics here called principal opponents analysis, and I commend you for that.  These are clearly stories about respected researchers doing very high quality, rigorous research that triggers debate, invites replication, and even in some cases, which I think is true at least in the case of Witte v. Peterson, the gradual acceptance of some common ground findings as the basis for collaboration and continued knowledge accumulation.  </P> <P>So that suggests for me two hypotheses that I would offer to Jeff and Andy that they may want to consider as friendly amendments in their papers.  The first is: if one measure of the maturity and quality of the scientific field is the extent to which scholars apply rigorous methods and models, publish their results, and argue the implications of their findings, it would seem that the awful reputation of education research is indeed not deserved, at least not uniformly.  </P> <P>The second is: if high-quality education research produces conflicting and controversial results, it is more interesting to explore the conditions under which those results find their way into policy discourse and become the basis for actual policy improvements.  I m suggesting here that if education researchers are frustrated by the slow pace of adoption of their findings, let me remind them all, let me remind us all that we are in good company one of the very first recommendations of the National Academy of Sciences in about 1865, I believe, was that the United States convert to the metric system.  Although that report has now cleared review, it has not yet been implemented.  </P> <P>Third point is about what might be called the cognitive psychology of the politics of knowledge.  That is a mouthful and I know Rick will take care of this in some blog or another.  I suggested earlier that even if research is good by some objective standards and the findings are even considered to be fairly robust, those are not sufficient conditions for their acceptance and adoption, and I just want to suggest that there is a macro-level analogy to something that people who study individual-level decision-making have known for a long time.  And that is even if one has very intimate, sophisticated knowledge of certain processes that does not lead necessarily to rational behavior.  </P> <P>One of the great decision theorists and probability risk analysts of our century, as I was told this story, refused to leave his home in the suburbs in Washington until they caught the sniper.  Now when you think about that, it is very interesting because this gentleman actually knew all of the statistics associated with rational decision-making here and yet decided to stay home.  Of course, after they caught the sniper, he went back on to the beltway, instantly increasing his chance of violent death about a hundredfold.  The point being rational thought does not necessarily produce rational behavior.  </P> <P>My final point is about rhetoric.  For reasons that would take a little too long to elaborate here, I think the condition of education research is subject to a similar amount and intensity of rhetorical excess as the condition of education itself.  </P> <P>I was not going to get into this except that Jeff reminded us in his paper of Rick Hess s post-AERA catharsis, which, for anyone who has been around long enough must have looked and sounded eerily like the good old days of Senator Proxmire and his Golden Fleece Awards.  Just to remind us that although the sex life of the screwworm did not sound like a particularly promising or smart use of public funds, even the good Senator acknowledged after giving that research project a Golden Fleece Award that the study actually had been a key to unraveling some of the major biological aspects of pest control and ultimately contributed to major gains in agricultural safety and productivity.  </P> <P>It is just too easy, even now it is obviously a lot of fun, to ridicule research just because of its goofy titles but it is not scientific, Rick, and had you sampled from the AERA catalogue a little bit less randomly, maybe a little bit more randomly, you might have noticed a large number of sessions dealing with evidence on the effects of high stakes test-based accountability, the implications of brain science for education, the cognitive basis for science education reform, just to name a few.  </P> <P>So, to conclude, which I noticed our time-keeper is warning me about, let me quote something that Lawrence Cremin of blessed memory, a great historian of American education, once wrote about education and then I m going to ask you to substitute in where he had the words education and schools, substitute education research.  Cremin wrote, If there is a crisis in American schooling, it is not the putative mediocrity and decline but rather the crisis inherent in balancing tremendous variety of demands Americans have made on their schools.   Substitute education research for schools and schooling and I think we will end up with a healthier, more useful, albeit perhaps a little bit less dramatic place in which to explore and ultimately improve the quality of our craft.</P> <P>Frederick M. Hess:  Thank you, Mike.  Andy, Jeff, any initial thoughts or comments before we open it up?  </P> <P>Andrew Rudalevige:  Just very quickly on the awful reputation issue. I think it is a fair point.  The title has become almost a substitute for the paper.  Though having read the paper recently, I can say that there is a certainly a comprehensive treatment of some of the reasons for that reputation whether I would agree that Dr. Kaestle does not think that it is entirely deserved.  As you know though, it is drawn from a much larger set of oral histories for which its own title becomes useful marker perhaps.  </P> <P>Everybody has been to fourth grade as the title of his larger work which draws together a lot of very valuable oral history.  It is a fascinating piece but the notion there is Yeah, everybody has been to fourth grade therefore everybody knows what needs to be done in education.   This gets to some of the comments that were made about well, if it is so easy to do, why should we bother to do it in a rigorous way and I think that is worth just adding that to the mix.  </P> <P>Frederick M. Hess:  Jeff?</P> <P>Jeffrey Henig:  Just one quick point for now also on this issue of the actual quality of research versus the uses of it in debate.  In the I m doing this -- both being embarrassed by being a shameless huckster, but in terms of my forthcoming book but just to show that I m not very good at it, I still do not have a title.  But in the book, I m looking very much at the uses of research and public debate but my argument is that the research itself in charter schools, despite the polarization in the public debate has been converging on a number of findings that are not consensual but are increasingly accepted and that research is working in arc pretty much like we would like to see it which is we know a lot more about charter schools now than we did 10 years ago but in the public debate that is not reflected.</P> <P>Frederick M. Hess:  Jeff, I wanted to ask you one of Gina s points was the acceleration of technology and the implications for communication.  It strikes me that it seems reading your paper that you actually seem to have a different take on that that you seem actually concerned that that accelerating communication can erode the protections and the insulation of the academy.  So what some of us see what some might see is anachronistic, you seem to see as actually the unique strength of the academy.  Could you talk a bit about that?  </P> <P>Jeffrey Henig:  Yeah, well, I m not a historian like others at the table so I can only reflect in an amateurish way on the folly of arguing against the onslaught of technology.  Things are going to change and I know they are going to change and you do not know how exactly that is going to play out but I do think without overstating it, without seeming like a crusty academic, which I am, I do think that there are some advantages in broad public knowledge accumulation to occasionally slowing down, taking a beat, waiting for more studies to accumulate, seeing where the center of gravity in multiple conflicting studies are, learning what replication shows and does not show, and that that part of academia is of value and could save us from a lot of hyper-flip-flops on issues as we cycle back from this-is-great, this-is-bad.  What do we know?  We do not know anything.  </P> <P>Frederick M. Hess:  Okay, let s open it up.  We should have two microphones out there.  Please catch Morgan s eye over here or Shane s eye over here.  Do we have a first question?  Susie?  And again, be kind enough to identify yourself by name and affiliation.  </P> <P>Susan Sclafani:  Susan Sclafani.  Chartwell Education.  I just wanted to pick up on that last point because when Gina was speaking I was thinking not just of technology in the sense of dissemination of research but the changes that technology is making in the way in which we are educating children, whether it is the concept that Negroponte has that give a child a laptop and they will teach themselves, or the use of simulations and virtual environments as a way to engage children in probably the most constructivist view of learning that we have ever seen as opposed to looking at direct instruction versus a little bit added-in discovery learning.  </P> <P>And I think the point that Gina was making is we are not doing any research it seems in those areas, though Michael s point about some of the sessions at AERA are getting at that a little bit, but where is the field in terms of looking at what is coming as opposed to what has been the dominant forms of education? </P> <P>Frederick M. Hess:  Gina.</P> <P>Gina Burkhardt:  I have this vision that the research community has started a lot of these conversations about what works and what might work under specific circumstances and then if we apply some of this to policy recommendations or policy changes, it actually could change some of the structures, the continuous learning of kids so that we are not confined with day-long, four hours or six hours or whatever.  Within that, then, the second level comes with the connection of practice to research.  They actually can implement some pilots where you watch how technology or how changes in structures really affect and the implementation side and the development side, it is not funded anymore, it is not happening.  So I worry like you, Susan, that we are moving to a place that we do not know is coming and we are not prepared in terms of collecting evidence or information without a future, rather a past.  </P> <P>Frederick M. Hess:  Jim?</P> <P>Jim Kohlmoos:  Jim Kohlmoos from the Knowledge Alliance, which is formerly called NEKIA.  I appreciate Gina s comments too, particularly folks in innovation and what Susan just said perhaps R&D becoming a stimulus for innovation, looking ahead rather than looking backwards, I want to ask Ellen though, looking at the history of education research, whether you think that there could be this potential shift towards the futures and R&D in education becoming an incubator for innovation.  Is that possible now based upon what you have seen in the past?  </P> <P>Ellen Lagemann:  Well, my first answer is I have no idea.  It is a very interesting question.  I was listening to Susan and to Gina and remembering and I m sure there are people in this room who will remember this better than I but there was a movement in the social sciences -- was it in the 80s? -- about social forecasting.  There were big issues of datalys and various things that were notoriously, I think, off the mark.  It is incredibly difficult to do but I would rather take up the point that Michael was making about you know, you can see this glass is half full or half empty.  </P> <P>I happen to share the opinion that I think he was putting forth and he can correct me if I m wrong that there is a lot of good work out there and generally it gets tarnished with the paintbrush of what was your example?  The screwworm or whatever it was.  So I guess, I m not If you look at the history, if you look at the founding of the Department of Education in the late 1850s and you look forward over time, I would say things are better but that has to be modified here or there caveats need to be added to the effect that there is so much more going on and it is spreading.  Jeff s numbers about the dissemination outlets is staggering and we have no standards in this field.  </P> <P>I remember when I was president of the National Academy of Education trying to work with AERA and how do we do something about this?  I think that is a huge problem.  And authority and power is so dispersed in education, we have such a chaotic, even an archaic system of governance in education.  If you look at all the local levels, let alone work that goes on in the federal apparatus, I do not know how we get our hands around that.  </P> <P>Frederick M. Hess:  Mike, your points before are being interpreted are you not concerned of the quality control today in education research?</P> <P>Michael Feuer:  No, no, I do not want to be misinterpreted as saying that everything is fine and we should go home, there is a I think an incredible amount of very healthy introspection on the part of the education research community about quality.  Look, if you pick up the latest bulletin of the American Economics Association annual meeting, I bet you could come up with just tons of really, really riveting examples of research that very few people would either understand or think has much practical application and yet economics as a discipline, as a scientific community is considered -- whether you like it or not the jewel of the social sciences.  </P> <P>So all I m suggesting here is that the education research community, which by the way has to its credit the development of a number of findings and methodologies and results that have been adopted by many other fields, should be grateful that we are doing all of this kind of discussion and constantly thinking about how we can continuously improve this craft.  </P> <P>The example I have in mind here is of course it is always funny to me that the people who are most interested in the so-called medical model are not exactly remembering that one of the great contributions to medical research - in epidemiology anyway - is meta-analysis.  Where do you think that was invented?  That was invented in education research and we do not get enough, we do not take enough and we are not offered enough credit for some of the very, very fine work that they takes place in education research.  </P> <P>Now, that said, this is not a defense for the status quo or an apology for all the shall we say, postmodern incomprehensibility of some research that goes on but I do think that we tend to undervalue ourselves perhaps a little bit more than we should, which goes along with the cycles of despair and exuberance that we feel about education itself and a little bit calming of the passions here would not be a bad idea.  </P> <P>Ellen Lagemann:  I just wanted to add one thing to what my pal here said.  Economics is a discipline.</P> <P>Michael Feuer:  Uh-oh.</P> <P>Ellen Lagemann:  Economics is a discipline and education is a field of research.  It is a field of practice.  It is a field of research and I think I happen to think that people who are involved in education research I do not care what discipline you are from or what school or institution you are sitting in I think all of us have something of a special burden, which is the end result ultimately somewhere down the line, you have to remember there are kids in classrooms who need to learn to read and count and think and it is hugely important for all of us, so I just think that it is hard sometimes to compare a discipline, where elegance in a tenure review can be sufficient, to some of the other fields.  Michael and I have talked about this, not 24/7, but often.  </P> <P>Gina Burkhardt:  So I think that comes back to the conversation about the community of contract researchers.  I think that was what Jeff called them.  People outside the academic research community that are now being brought into the mix of doing research on a market-driven, client-focused basis and while the community can have these arguments about rigor and relevance, they may not be addressing they may be but they may not be addressing the needs that have been identified by the policy-making or practitioner community just as you suggest, Ellen.  </P> <P>So that, I also wanted to take a little bit of debate around the idea that these contract researchers methodology may not be as rigorous or as relevant when I think it is, and I m not sure that it is as biased as you indicate in your paper in terms of being pushed to have the right answer to the response that a contractor wants but rather that they are available, they are affordable, and that they are willing to be client-focused and market-driven. </P> <P>Frederick M. Hess:  Jeff, do you want to speak?</P> <P>Jeffrey Henig:  I ll do it but I m going to indirectly respond to Gina, but not directly and I m going to do it by starting with Susan s question about a particular research area and my reflection on that was if we went around this room even and then recognizing that education policy is made here but it is also made in a lot of other places and went around those rooms too, the number of researchable questions that are important is huge.  And I start to think but the way we tend to deal with this in Washington is to elevate two or three or four of those at a time and try big banged-up studies.  </P> <P>Ultimately, I think, my way around this and this is my indirect response I personally feel the more research, the better and that the only way we are going to address many needs, especially when we cannot predict what is coming down the road is have a lot of folks doing studies, some of which people are going to think at that time are a waste of time and it is going to turn out some more of it.  </P> <P>To me, if that is right, if what we want is a multiplicity of studies, we are concerned about quality and sometimes there is reason to do big, big expensive studies but what we want is a multiplicity of studies.  There are policies that would work to build the infrastructure of research, that reduce the marginal cost of research and make it more possible for people to do studies and to link together on scaling up issues to similar databases.  </P> <P>I m going on too long but I ll just wrap it up.  </P> <P>Frederick M. Hess:  -- Jeff, wait.  Give an example, for instance.</P> <P>Jeffrey Henig:  Okay.  So I pointed out one of the problems with scaling up and with fragmented research is that people are making up their variables, making up their measurements all the time because there has not been good data.  Some of that is beginning to change.  It is also a technology issue but as the states are building these administrative databases, student level data  for all its limitations NAEP and NELS and some of these others where there is a common research where the cost of building the data infrastructure, at least the key deep-handed variables, the outcomes that we are interested in has been collectivized that makes it a lot more possible for researchers who may not be highly funded, who may have somewhat idiosyncratic research questions to come and do those studies and I think that is a plus, it makes studies that are done in Texas more relevant to studies that are done in New York.  I think we are moving in that direction but I think we could be more self-conscious about it.  </P> <P>Gina Burkhardt:  I agree.  </P> <P>Gregory McGinity:  Hi, I m Gregory McGinity with The Broad Foundation in Los Angeles.  Mike, you talked a little bit about the importance of reputation and Jeff, in your paper, you talked about one of the things that I think is really interesting is that unlike in medicine where you have JAMA and other prestigious journals, there is sort of the one everybody looks at, we do not have that in education.  Is one needed and could you actually get to the point that one was developed and do you think that would change the way the field was viewed?  </P> <P>Jeffrey Henig:  Well, see, I can predict now that Ellen will after I say this remind me that education is not a discipline, it is a field and so it would be hard to do it.  But I do think there would be an advantage in having a clear hierarchy of journals within the education arena and a high-quality journal of the JAMA status that publishes really good stuff.  </P> <P>I have talked in my research with journalists and a lot of times, they feel a disadvantage by not having an authoritative checkpoint on quality that they can rely on.  I think it would be possible to do it, to write it in a way that is not down to the public but is meant for audiences beyond the experts and would have to probably have a speeded-up peer review process in order to keep a certain sense of currency, but I think that can be done if people are recognizing that this is the most important journal or the most important couple of journals in the field. </P> <P>How do you do it?  I do not I have thought a little bit about it I do not know who has the incentive to fund and create such a journal and make it happen.  I see it as a public good in the classic sense.</P> <P>Frederick M. Hess:  Did you want to stand again?</P> <P>Ellen Lagemann:  I just want to add just one thing very quickly.  I was on a committee of the National Research Council that looked at some of the issues around journals and I think I do not remember the report well enough to remember the details of it.  I m sure it is available.  But we certainly looked at journals and thought about what their impact could be and sadly, I failed to do anything about this, the most read journal is the Harvard Education Review, which is not peer-reviewed.  It is a student journal and the students are the ultimate arbiters of what goes in there and that is a huge problem, I think, for the field.  It is absolutely wonderful to have students learning to do all sorts of things but in my view it would be very complicated but it certainly worth thinking about.  </P> <P>Jim Farmer:  Jim Farmer.  Georgetown.  In the past month, I have been working with education technologists in the UK.  One refers specifically to the IES report on education technology and Ellen, I think this one will straighten your point, the </P> <P>Frederick M. Hess:  Would you speak up a bit, please?  Thank you.</P> <P>Jim Farmer:  I think this would illustrate Ellen s point about practitioners.  None of them had read the IES report but everyone knew about it from either a blog or from an online description of it and their conclusion of it was that the US has decided that education technology is not significant and no investment should be made in it.  What do you do about this question of dissemination? </P> <P>Ellen Lagemann:  I m going for two of it in just a minute.  I m not totally clear to what your question is.  Are you talking about what should people involved in education research be concerned about technology or people in the practice of education?</P> <P>Jim Farmer:  There is a difference between providing information to the practitioner versus writing a journal for tenure.  </P> <P>Ellen Lagemann:  Yes, yes, yes.  Well, my own personal view is we have not even begun to understand the processes that need to be involved to translate the findings of rigorous scientific research into application, whether that application is clear recommendation, it is for policymakers that are backed up with the kind of backing up that they need to have, or whether it is the little widget that can help Johnny learn to read.  </P> <P>So, I do not think we have an understanding of the engineering involved, let alone will to do this.  We seem to be more interested in generating knowledge than in worrying about its use, which is why this conference is so important, I think.  And to me, I have spent my life in the university.  I deeply believe in research.  I believe in research which has a value even when you cannot see its immediate value.  I do not mean to be denigrating that but I do believe that use in a field like this is hugely, hugely important.</P> <P>Andrew Rudalevige:  I was just going to say having the Clearinghouse which evidently is not being looked at as directly as was hoped, was a very self-conscious effort to disseminate that knowledge in a way that could be translated into little green and red colored symbols.  It is a very simplified effort.  Well, this one gets the seal the Good Housekeeping seal this one does not.  This one works.  This does not.  To the extent that is not being utilized by the education research community, that is, I think, going to be very problematic.  I do not know what the answer is.  Maybe somebody from IES here can talk about how they hope to build on that.</P> <P>Frederick M. Hess:  Gina?</P> <P>Gina Burkhardt:  I once heard Michael McPherson speak eloquently on this exact subject about translation and whether or not that was the right word.  Maybe when he is up here, you can ask him that question.  </P> <P>Frederick M. Hess:  Okay.  Yes, sir?  Shane?  Oh, there we go.  </P> <P>Larry Snow:  Larry Snow with Houghton Mifflin Company.  Following up on that question, there is a translation of the research, not just to the research community and the policymakers but the intermediary of the media and as you have discussed the media now is much broader through technology like blogs and immediate discourse, if you will.  At the same time there are fewer education writers.  There are fewer bureaus in Washington and so there is also a constricting of some of the outlets and in talking about translation of research, there have been distinctions between the research and the press releases that came out of the Department of Education and the nuances of for example, the educational technology study as a first year study and many other limitations get lost.  </P> <P>And we are also now in an era where there is greater media attention to research.  Again, the educational technology study gets the front page in New York Times.  And then a follow-up of school districts dropping one on individual computers and so there is the question would be the responsibility or the awareness of the researchers and how the research gets described for broader public consumption.</P> <P>Jeffrey Henig:  Well, I think these are really very important aspects of what I would refer to as the ecology of research use that go well beyond first order approximation that research is lousy so that is why we do not see more of it in practice.  I think what you are getting to here is that this is a very complex story and there are these intermediate organizations that for better or worse are doing some form of I do not really like the word translation either, but they are certainly distilling out of scientific studies what they perceive to be the nuggets of knowledge that may be most applicable to specific decisions that people want to be making better.  </P> <P>That said, again, I think we just need to recognize a couple of things that even when the research is pretty much above reproach now, look, I do not want to get into a whole big fight here and but most of the scientific community, the respectable scientific community has pretty much figured out that evolution is where it is.  This has not made it </P> <P>Frederick M. Hess:  -- do not go and get controversial with us now.</P> <P>Jeffrey Henig:  Well, but I just want to suggest that no matter how well you translate the theory of evolution, you are not going to necessarily convince various school boards in the country that and you know where I m going with this.  So I think, with respect to some of the work we do, there is a big opportunity here for setting our own standards and setting our own sights on a somewhat more gradual, incremental appreciation of what this research means and getting away from the Here s the result.  Why don t you knuckleheads in the government just apply it?   Because that is not the way the world works.  </P> <P>There is a not to I have historians all around me here, but I think it was in Piers Plowman, which is a 14th century bit of English literature, that the first reference comes up to the phrase L homme propose, Dieu dispose, which means Man proposes and God disposes.   And you find periodic reminders of this all through history.  I would offer as a metaphor for what we are talking about that Science propose and politique dispose.  With that I think we can have, maybe have a better mutual understanding of what is possible in this field.  </P> <P>Frederick M. Hess:  Andy.  Jeff?</P> <P>Andrew Rudalevige:  Well, I was just going to say but he beat me to it.  I thought it was The president proposes and the Congress disposes.   One thing I think that the structural debates here may give education research the time to do is give us some space and to back off the demand for immediate results with the possible and optimistic result perhaps that these big questions about the conditions under which things work can actually be explored in some detail.  I tell my research methods classes and I do want to thank my colleagues here for not picking on political science as they well could have and when they do turn to economics instead, it was a in any case, I tell my research methods classes that there are no interesting yes-or-no questions, in fact.  </P> <P>All the questions come to it depends.   That is the answer but that is not the last answer and that answer in the final exam is not going to get you very far.  What will get you further is the idea of what it depends upon and when and so I think it comes back to Gina s point about the notion of conditionality and the space we might be able to give the research community to get that. </P> <P>Frederick M. Hess:  Well, I think that is an outstanding closing note for this first panel.  What I hope we have done effectively in this panel is get on the table a number of the questions that we want to come back to during the day.  On the question, for instance, of what is to researchers obligation to ensure that the work is represented accurately, I hope that everybody will push Richard on this in the next panel when Richard Ingersoll is up here.  On these questions of disseminational blogging, while Andy Rotherham is up here talking about school choice, I think, talking about these issues and the dissemination of research and analysis will be entirely appropriate.  </P> <P>I think Michael, as always, summarized nicely what we, I think, want to keep focused on all day: is that regardless of the quality of research, rigorous and relevant research is not enough.  It is not clear, necessary or obvious that it will be used even when it is available.  The real question for today is how does one increase the likelihood that when rigorous and relevant research is available that it will be used in a way which we think is sensible and appropriate and we will come back to that.  We are going to take a ten-minute break and then we will start with the next panel.  Thank you.  </P> <P> </P> <P>[Start of Panel 2: How Research is Used-Teacher Quality & Reading]</P> <P> </P> <P>Frederick M. Hess:  Let s all go ahead and get seated, please and get started.  Okay, let s all go ahead and take our seats, please.  Okay.  Take another one or two moments but let s go ahead and get started.  Alright, we are going to go ahead and get started.  Well, this is the second panel of the day, the panel which is going to look at how research is used in the case of teacher quality and the case of reading.  We have with us two authors and two discussants on this panel.  </P> <P>Speaking first will be James Kim.  Jimmy is an assistant professor of education at the Harvard Graduate School of Education.  Jimmy s research interests include the use of quantitative methods to assess the effectiveness of compensatory education policies for disadvantaged students and the impact of reading programs on adolescent learning.  </P> <P>Speaking second will be Richard Ingersoll.  Richard, a former high school teacher, is currently a professor of education and sociology at the University of Pennsylvania.  Over the past decade, Richard has done extensive research on the problems of teaching shortages and under-qualified teachers.  His research on these issues has been widely reported in the media and featured in numerous major education reports.  </P> <P>Speaking third will be Lorraine McDonnell.  Lorraine is a professor of political science at the University of California at Santa Barbara.  Prior to joining the UCSB faculty, Lorraine was a senior political scientist at RAND for 16 years where her research focused on the design and implementation of education policies and their effects on school practice.  Lorraine is also, I guess, in about a year, the incoming president of the American Educational Research Association.  </P> <P>And finally, we have Reid Lyon.  Reid is Executive Vice President for Research and Evaluation at Higher Ed Holdings and Best Associates and Whitney International University headquartered in Dallas, Texas.  Prior to joining Best Associates, Reid was a research psychologist and the chief of the Child Development and Behavior Branch within the National Institute of Child Health and Human Development at the National Institutes of Health or NIH.  With that, Jimmy, would you please get us started?</P> <P>James S. Kim:  Good morning, everyone.  My name is Jimmy Kim and the title of my paper is Research and the Reading Wars because -- I do not know why.  I have a fairly straightforward agenda.  I want to do four things in my talk today.  First is to answer this question: What is the Reading War about? Two is: How has the war been fought? Three is: What are the consequences for the research and policy and practice community?  And then I want to conclude with some suggestions rather than recommendations, because I m not sure how good they are and I would love to get feedback on it.  </P> <P>So, what has the Reading War been about?  My sense in reading the history is it is really about two things.  One is a fairly specific question about the efficacy of different types of instructional practices in beginning reading.  And Jeanne Chall, in her classic book Learning to Read: The Great Debate in 1967, basically summed up the question quite well.  She said, Do children learn better through a code emphasis where teachers instruct children about the specific relationships between letters and sound, or do they learn better through a meaning-based emphasis?   And the strategy that she specifically looked at was look-say, just teaching children to look at whole words and then to say them and to focus on meaning.  </P> <P>But I think the second thing that the Reading War has been about is this question about professional autonomy.  That is, who should govern curriculum and instruction in the classroom?  Frank Smith, who is a whole-language leading theorist, started writing in the 70s about the importance of having teachers be the ones who make these curricular decisions.  Since the Reading War is a debate, I think, about an instructional question and a political question, it is not surprising that the war has been fought in two different venues.  To address the instructional question, what scholars have done, like in other fields of education where there is debate, is to conduct experiments, publish in peer-reviewed journal articles and then, occasionally, every five to ten years, we convene expert panels to tell us what the research says.  In reading, the theoretical debate has focused on this question of Does context help children to decode new words?   If context helps, maybe we do not need to focus so much on letter-sound relationships.  </P> <P>And so what we see starting in the late 1960s after Chall s publication is a series of scholars who debate as adversaries and publish in these journals.  So we have Chall saying that In looking at 30 experimental studies from 1900 to 1965 comparing code and meaning-based strategies, 27 favored a code emphasis.   Two years later, Ken Goodman publishes an article saying, That is actually incorrect.  Children seem to read words quicker in context and sentences than in isolated lists.   Then in the 1970s, this debate is picked up by other scholars until there is a scientific consensus that starts to form.  We have been very involved in this research and we begin to resolve some of these questions.  So, that is just one way that the war has been fought and the importance of this venue that is typically used to adjudicate disputes in science.  </P> <P>To promote teacher autonomy, the whole language theorist at the same time disseminated ideas through professional organizations and state local boards of education.  And what I talk about in my paper is that this debate has been a political debate about whether policy mandates promote teacher autonomy.  And so whole language theorists started communicating directly to teachers arguing for the need to change practice and, most important of all, arguing that teachers should be at the vanguard of policy making, and it should not be driven entirely by results published through peer-reviewed journals.  What are the consequences of fighting the Reading Wars through these two venues?  </P> <P>This is an actual reviewer s reaction to a paper that a very distinguished literacy scholar had submitted to the Journal of Reading Research Quarterly in the late 1970s.  And you see kind of the Reading Wars being fought through this journal, and I m not going to go through all of these comments; you all can read it.  But I will go to the last comment on suggestions.  This is what the reviewer said: When will we get to the real issues?  When will we try to look at real kids reading real language and when will we lift our eyes from the word to meaning?   So, this theoretical dispute is being fought in this peer-reviewed journal, and this has a couple of very important consequences.  </P> <P>One is that, as we all know, peer review is a very slow process and evidence accumulates in a very evolutionary fashion.  And so, the consensus about early reading instruction and what is most effective begins to build in the 80s and 90s and it is not surprising that the two most influential expert panels occurred in the late 1990s, 20 years after these theoretical disputes were first fought over in journal articles.  And what is important about this process is that scholarship what makes ideas valued in the scholarly community is that findings have to replicate over time, over different laboratories, over different samples of students.  And a basic idea is that no single study or single scholar should ever dictate policy recommendations; it is a democratic enterprise where it is the accumulation of findings.  </P> <P>A second consequence of working through peer-reviewed journals is that research is almost always one or two steps behind the demands of practitioners.  Like if you are a teacher, you have to fix reading problems in your classroom today; you do not have 20 years to wait.  If you are a principal, you have to worry about making AYP next year.  You do not have 20 years to wait to see what the researchers have to say about children s reading deficits.  </P> <P>But a third important consequence is that expert panels typically do not include teachers in the policy-making process.  If you look at the 1983 Becoming a Nation of Readers report which was issued by NIE and the 2000 National Reading Panel report, there is one practitioner, and over 10 university researchers.  </P> <P>What are some of the consequences of working through professional organizations in state and local boards of education?  One consequence, I think, is that whole-language theorists were very effective in disseminating their ideas by directly appealing to teachers and professional organizations like the National Council of Teachers of English, and so the practice is scaled up very quickly even in the absence of evidence.  In 1987, if you look at the California State Curriculum for English Language Arts, it is very receptive to whole-language practice.  And a 1987 survey of 43 textbooks used to prepare teachers suggested that whole-language had become the conventional wisdom.  David Pearson who is the dean of the Berkeley Graduate School of Education said this in 1989: Whole-language has spread so rapidly throughout North America that it is a fact of life in the literacy curriculum and in research.   </P> <P>But I think there are a couple of important unintended consequences of this rapid scale-up of a policy that was not tested, I think, through a lot of experimentation.  One is that whole-language was very unresponsive to empirical evidence right about the time that it became the conventional wisdom.  So in 1989, there was a meta-analysis conducted by Steve Stahl, which suggested a code emphasis is superior to whole-language practices, echoing Chall.  But it is not incorporated into changes with practice.  A second important unintended consequence is if whole language is the conventional wisdom - it is what the states are doing, local districts are doing, what teacher education programs promote - that means that any decline in reading skills can be very easily linked to the conventional wisdom.  </P> <P>And so what happened in the early 1990s is that California underwent one of the largest declines in its fourth grade reading scores on NAEP.  By 1994, it was next to last and the decline was observed with all subgroups.  Well, from 1994 to 1997 California also reacted by issuing a series of mandatory phonics bills, and so did 33 other states.  So, this suggests to me that one of the unintended consequences is that external mandates passed by state lawmakers are essentially undercutting the professional autonomy that whole-language theorists had hoped to secure in their movement.  </P> <P>So, I want to conclude with a couple of thoughts about where we are with research and some recommendations.  I actually think, today, the science is quite good.  We have been conducting these experiments for 30 years.  The National Reading Panel really got us beyond the dualisms of whole-language versus phonics, and my reading of that report is that it is very pro-teacher.  It essentially said teacher-directed instruction is very effective in improving student outcomes with respect to word-level, sentence-level, as well as text-level comprehension.  But, more importantly, it also said the one strategy where teachers play a minimal, if any, role -- sustained silent reading had little or no impact on student outcomes, that direct teacher instruction is very important.  </P> <P>But I think that the major problem in the Reading Wars is not the science; it is political.  One is that there are adversaries who continue to argue back and forth about retrospective interpretations of the research.  So even with the National Reading Panel report there are very distinguished scholars who argue that the criteria that was used to screen studies was too narrow, and there are too many studies that were not cited; it was a very selective interpretation of the research.  </P> <P>A second political problem, I think, is that teachers are excluded from the decision-making process.  There was a practitioner on the NRP who issued a minority view, which was on the last two pages of the report, by the name of Joanne Yatvin. And this is what she said and this is a criticism we hear over and over through these expert panel reports.  She said, The university researchers were unqualified to be sole judges of the readiness of implementation in the classroom of the NRP s findings.  And further, outside teacher-reviewers should have been brought in to critique the panel s conclusion just as outside scientists were to critique its processes.   </P> <P>So I want to conclude with some suggestions.  I think in reading as in other areas of education policy, the formation of what I m going to call teams of rivals in the creation of adversarial collaboration should become the norm.  I take the title Teams of Rivals from Doris Kearns Goodwin s history of the Lincoln administration where she points out that President Lincoln made the very unusual decision of choosing his former political opponents to become members of his Cabinet.  </P> <P>This is why I want to conclude with these two points.  If the Reading Wars is partly driven by debates concerning political power and involves legitimate controversies about theoretical issues, such as the role of context in word identification, then I think we need to think of ways to deepen support among the key constituency teachers who are being asked to carry out the reforms.  And we need to deepen the legitimacy of evidence produced by scholars who are engaged in very politicized debates, many of which Jeff referred to this morning.  </P> <P>So first this idea of expert panels including more teachers.  I draw an idea - this idea from the United Kingdom s Literacy Panel, which had an equal mix of teachers and researchers unlike the US National Reading Panel was charged with the task of making recommendations for improving reading in underperforming schools.  I think including more teachers on expert panels could deepen the legitimacy of research.  And I also think that research and teachers share a kind of what works epistemology and that their findings would converge and agree quite substantially.  The 1998 National Reading Council Report Preventing Reading Difficulties in Young Children was conducted -- the lead author of that report was Catherine Snow.  And two years after that report was issued, this was what she said about it.  She said, Nothing said in the report about reading instruction could not have been formulated by an experienced, thoughtful, reflective first grade teacher with a few weeks free time.   </P> <P>Second, if research gets politicized and is used selectively to support popular but untested reforms I think we need to conduct the research where adversaries collaborate and evidence has a potential to change people s minds.  So I want to conclude that this last idea in psychological science where they outlined this procedure called adversarial collaboration.   This procedure requires antagonists to collaborate on a prospective study, agree on an arbiter who imposes the rules of engagement over the process.  The arbiter helps the adversaries decide on the design of the experiment, controls the data, determines the final venue for publication and can even declare in the final publication if a scholar was uncooperative.  Dissemination of these findings is much faster than what we typically find in these peer-reviewed journal debates that go on for decades.  I think such collaborative efforts would help resolve some of these education controversies and, more importantly, it might actually deepen support for facilitating the translation of good science into practice.</P> <P>Frederick M. Hess:  Thank you Jimmy.  Richard.  And for those of you who are interested in reading issues, while we get Richard up on the screen here, I think Jimmy has really rendered a really comprehensive treatment of a lot of these issues for folks who have wondered exactly how the Reading Wars unfolded.  So again, Jimmy s paper, along with all the rest of them, is available out in the foyer.</P> <P>Male Voice:  Thank you.</P> <P>Frederick M. Hess:  Well, let me actually preface this by saying that Richard s paper is a bit different from the others because I actually asked him as somebody who has been in the middle of one of the ongoing policy debates of the past decade to write from a first-person perspective about it.  So, it is not Richard kind of going off on an ego trip so much as me commanding Richard to go off on an ego trip.</P> <P> Richard Ingersoll:  Yes, it was a very unusual task and it is a slightly different topic and talk I m going to give, perhaps, than some of the other papers.  It is Rick -- my marching orders were to tell more of a personal story of this one line of research I have been doing over the last decade and a half, that is, the problem of under-qualified teachers and how my data and my research were, and were not, used in the policy arena.  In some ways, I need to say that it is not an entirely happy story from the viewpoint of data based decision-making and the rational use of research in policy formation.  And in some ways it is a story of, I think, where the data were really used and are used to buttress positions that are really wrong diagnoses and the wrong prescriptions.  </P> <P> But, first, I want to just backtrack a little and give a little personal history because it shapes how I came into this particular line of research.  I m a former high school teacher and one of the most frustrating things I ever encountered on the teaching job, something that I never knew ahead of time that I would encounter, was this odd phenomenon called out-of-field teaching where you are, once on the job, assigned to teach things that you really do not know very much about, and that do not match at all, or, maybe, only match a little bit, your preparation ahead of time.  </P> <P> So I was a Social Studies teacher and I taught American History and World Problems and things like that, but there really was no semester that went by in my teaching career, both public and private schools, in which, in addition to my Social Studies courses, my beloved courses, I was not also assigned to teach -- you name it:  Special Ed, Algebra, Sex Ed, all kinds of things that I really did not know anything about.  </P> <P> Needless to say, it is really challenging to do a good job teaching things that you do not know very well and it is very frustrating.  And so I started to puzzle.  Well, what is going on here?  Is this going on in other schools?  Was I just unlucky in the particular schools that I got a job in?  Does this go on elsewhere?  And if so, why?  Well, luckily, I got a chance to eventually tackle those very questions, those very issues.  And I eventually quit high school teaching and got a PhD in Sociology, and just as I was finishing up, voila, the Department of Education released this wonderful huge new data base on teachers, called the Schools and Staffing Survey.  And myself and a couple of analysts at NCES figured out -- for the first time ever, we could actually figure out how much of this out-of-field teaching goes on in this country.  </P> <P> Well, it was interesting; even I was surprised by the findings.  And I quickly discovered that my personal experience really was not unique at all.  Indeed, we had a national problem on our hands.  Wait a second, okay.  This is just a sampling of the kind of things we found.  Just to give you a quick taste, I will run through this very briefly.  These are national data and the secondary-level teachers, 7 through 12 in the core academic subjects.  </P> <P> And the bars simply tell us the percentage of teachers across the country who are teaching in a particular field that do not have a college minor or major in that particular field or something related.  So, for instance, the second bar tells us that over a third of Mathematics teachers, people teaching Math in public schools across the country, do not have at least a minor or a major in Math or things related - Math Education, Statistics, Engineering, Physics, et cetera.  </P> <P> So, when we first started to release these data, they were alarming and they did capture a lot of attention; attention from politicians, attention from researchers, attention from advocacy groups, attention from the media, et cetera.  Myself, at the time, being a young assistant professor in the Sociology Department of University of Georgia, it is very exciting.  It is very gratifying to have your phone ringing off the hook and everyone wants your data and wants to know all about it; that is a huge ego trip in a sense.  But it also turned out to be a very frustrating experience for me and still is, in many ways.  </P> <P> What I found out quickly was not entirely, but most of the time, the data were misunderstood and misinterpreted and often misrepresented.  And this misunderstanding always seemed to revolve around what I felt was the crucial question, the Why question:  Why is so much of this going on?  How could it possibly be that over a third of the Math teachers in the country do not even have a college minor in Math or something related?  That is not high bar.  How could this possibly be? </P> <P> Very quickly, two views dominate, and continue to dominate, this question and this issue, two theories, really, as to the Why question.  The first one is that teachers are at fault here or, more specifically, their education, their training, their preparation; that the source of out-of-field teaching lies in inadequate schools of education, inadequate training and education preparation, et cetera.  </P> <P> Now there are different variants of this view depending on sort of where you fall in what we might call the teacher quality wars as opposed to the Reading Wars.  One of the most popular variants -- its interpretation was that, really, the source of this problem is that not enough of these teachers are required to take subject matter courses; they take far too many of these soft pedagogical education school courses.  </P> <P> And the solution, really, is to require teachers from now on -- a solution to this problem -- require teachers to take a real degree or major in academic discipline - History, English, Mathematics, et cetera.  Now, this maybe a very cogent position but it actually is totally incorrect, this view, when it comes to out-of-field teaching.  The data tell us that, in fact, all teachers have a college degree; half of them have master s degrees and, contrary to popular wisdom, over 90 percent of teachers have full-teaching certificates.  </P> <P> Yes, it is certainly true that often there is a problem with the preparation of teachers and ed schools certainly can be inadequate; but that is a different problem.  The source of out-of-field teaching is not in the lack of preparation on the part of teachers; it is in the lack of fit between, on the one hand, what they took prior to the job and, on the other hand, what they are assigned to teach once on the job.  So then the question becomes Why is there all this mis-assignment?  Why is this so prevalent?   </P> <P> And this brings us to the second piece of conventional wisdom which is assumed by everybody and that is shortages - we have a shortage of teachers; principals cannot find adequate people to fill their positions; they have to put a square peg in a round hole.  Voila, out-of-field teaching.  This is a very reasonable explanation and, indeed, the data tell us there is a little bit of truth to it.  </P> <P> Clearly hiring difficulties in schools are a factor behind these numbers.  But there are a couple of glaring problems with this theory.  One, shortages cannot explain the fact that we have high levels of out-of-field teaching in fields like English and Social Studies that are surplus fields.  Secondly, the data tell us that about half of the schools in the country do not suffer from shortages.  They do not have hiring difficulties; they actually have waiting lists.  So then, why is it that we have this problem going on in schools?  </P> <P> This brings us to my own hypothesis and theory and counter to the dominant ones, which is that it is very simple; it is rooted in the way these places, these crazy places called schools, are organized and managed and run.  Principals, it turns out, make this key decision, as to who is assigned to teach what during the day.  And it turns out there are all kinds of reasons why a principal might find it quite effective, cheap, convenient, et cetera, to put a square peg in a round hole.  You are the principal; one of your Math teachers quits two days before the fall semester starts.  Now, you ve got a choice.  You could go to the board, get a new position, advertise at Hire Zone.  Well, then it is November before you have that class filled.  Or you could take a couple of these Richard Ingersolls or those other teachers in another field and mis-assign them for part of the day.  </P> <P> Turns out there are dozens, if not hundreds, of such managerial decisions that principals are forced to make in filling up with a limited staff and a limited day this wide menu of courses which the public demands.  And so we often get this problem.  And, in many case, we could argue that what the principal is doing saves a lot of time and money, certainly, from the principal s viewpoint and even from the tax payers viewpoint to mis-assign people.  But it is probably not a cost-free decision and with NCLB, with No Child Left Behind Act, it has now become an illegal decision.  </P> <P> Indeed, the legislation mandates in the core academic fields, teachers must be assigned -- all classes must be staffed by teachers who are qualified in the subject taught; that is the crucial language in the legislation.  So let me just quickly close here - I m sure my time is about up - talking about the policy implications for these different diagnoses.  </P> <P> So everybody believes that the source of this problem is a teacher deficit; it is either poor qualifications, poor preparation, bad ed schools, or it is too few teachers.  So, in other words, it is either a qualifications or a quantity deficit.  And so, of course, this leads to all kinds of antidotes.  But the data tell us that none of these are really going to work.  In plain terms we could bring hundreds of thousands of new teachers into this occupation; we could require them all to have a PhD and an advanced certificate.  That maybe a very worthwhile thing to do, but it would not solve this problem if large numbers continue to be mis-assigned to teach other things entirely once their on the job.  </P> <P> The data tell us that the problem is not so much out there in ed schools and supply; a lot of the problem is in here.  It is in the ways schools are managed and run.  That is a very different direction from what, basically, the conventional wisdom has been telling us, entirely.  </P> <P> Now, I have a project going on right now using the national data to evaluate, really, if NCLB has had any impact on this problem.  And so, we are looking at the data a couple of years before NCLB and then several years into NCLB.  And the data we found thus far - this is work in progress - are very revealing, and it is actually not a very happy story.  The data tell us that there has been very little change in the amount of this.  </P> <P> And my second slide here -- this is just an overall summary.  The top bar is the percentages of classes, core academic fields, secondary level, in which the teacher does not have a major in the subject.  And that is several years before NCLB; in the bottom bar it is several years into NCLB.  And, yes, the numbers drop to a couple of percentage points, barely statistically significant; very little impact on this problem.  Now -- but there is also something that is a little more optimistic in the data.  This is an overall average here.  The data also tell us that there are very, very large differences between different types of schools and between different states in the extent of this problem.  So, that raises all kinds of interesting questions:  Why is it that some schools - and this is after controlling for poverty - have so much less of this problem going on?  Why is it that some states have so much less of it than others?  </P> <P> So, my own hope is that this story is not yet over, that there is a definite role here for data and for research to keep plugging away, to try to actually counter the conventional wisdom which has dominated, and to try to get a better understanding and, perhaps, better solutions as to just why this problem is going on.  Thank you.  </P> <P>   Frederick M. Hess:  Lorraine.</P> <P>   Lorraine McDonnell:  There are many disadvantages to growing older but one of them is an enveloping sense of d�j� vu. And I must admit when I saw this subtitle to this conference Why education research does or does not influence education policy, I thought of past events.  So I remember the booklet that I had put together about six years ago for AERA members who wanted to try and translate their research findings into policy advice.  I remember the Social Science Research Council Conference that I attended about two years ago.  I remember Ellen was the keynote speaker, and it had the same subtitle but the title of the conference was Fits and Starts.   I m sure that many of you can think of other similar activities, thus, demonstrating that although education researchers fall short of the useful to policy standard, we are willing to admit our shortcomings and keep trying harder.  </P> <P> And I think that Richard Ingersoll s and James Kim s papers contribute to that effort not just by analyzing the uses and misuses of research on teacher qualifications and reading, but also illustrating some broader lessons about the connection between research and policy.  And what I like to do is comment on three of those lessons and then close with a brief suggestion.  </P> <P> The first comes from James Kim s paper and his portrayal of whole language researchers direct appeal to teachers and their professional associations, and then the eventual California Language Arts Frameworks.  In his paper, he contrasts that strategy with the experimental studies of the basic processes underlying reading that were undertaken by linguists and cognitive researchers.  The latter studies were in the mode of traditional social science research, basically understanding social processes and the relationships among them.  But this research is typically not done with policy makers as the intended audience, so it is rarely subjected to the kinds of financial, political and administrative feasibility analyses that we do expect of policy research.  </P> <P> The whole language research, on the other hand, turned out to be most people would say more in the realm of untested theory rather than validated findings.  But nevertheless, those researchers did understand the extent to which the research-policy linkage depends on policy entrepreneurs who are committed to an idea or a particular strategy and sufficiently skilled to move it through that fragmented institutional system which is our education system with its multiple points of access and networks; and I think James s paper shows that.  </P> <P> So, one implication or lesson I would take from his paper on the Reading Wars is that good social science research on the learning process is not a sufficient condition for policies.  That requires two other conditions; the first would be an analysis of those findings that are likely to be modified once you impose real-world constraints of limited resources, political incentives, and the bounded rationality of implementers.  </P> <P> And the second, then, is the availability of entrepreneurs, and these are people who can both translate social science research into policy research and then policy implications but also use it in linking Kingdon s problem policy and politics streams.  And I think it is interesting that in the case of reading research that policy entrepreneurship, eventually, happened through the consensus panels that performed the translation function and then also played a role in combining the three streams to create a policy window.  </P> <P> But I think there is also a lesson there and that is once you go to policy entrepreneurs, there may be a trade-off for researchers because some of the people who take up your research as entrepreneurs may, in the course of it, distort it or misapply it.  And I think that did happen, for example, in my native State of California when curricular and cultural conservatives took up the reading research to push on a broader social agenda; but, still, it does show the importance of policy entrepreneurs.  </P> <P> The second lesson comes from Richard Ingersoll s paper and his argument that neither the policy focus on teacher pre-service requirements nor deregulation of entry are the appropriate solutions to the problems out-of-field teaching; rather, it is to change administrative practices by which teachers are assigned to courses within schools.  Now, when I read that discussion I immediately thought of one of the first lessons we teach budding policy researchers, and that is that policy makers recognize that their leverage over what happens in individual schools and classrooms is very limited.  So they are most interested in research about those factors over which they have measurable influence.  </P> <P> Now, policy makers likely misapplied the research on out-of-field teaching because they can more easily control the requirements for teacher training and licensure.  It is much more difficult for policy makers at the top of the system to control or even influence school-level practice because in the case of teacher assignments, it would have either required allowing schools to have a lot more slack in their resources than is typical or having the sufficient capacity that the district or the State level to enforce course assignment standards.  </P> <P> So, a second implication I take from that paper is that, research that points to changes in well-embedded school or classroom-level practices may require a more circuitous route from findings to policy advice, often necessitating additional study that considers the organizational conditions and incentives needed to facilitate new patterns of administrative and teaching practice.  The third implication comes from both papers, and it is more a reminder than a lesson.  Both papers document what happens when policy makers, practitioners, and the public get in the grip of an idea.  Whether it is whole language awareness and phonics instruction, teacher professionalization or alternative routes to teaching, those ideas became the policy images that defined how particular policies were discussed and understood.  </P> <P> They combined information, albeit sometimes incomplete or misunderstood, with the ideological and a mode of appeals.  Now, we all recognize at some level that ideas are critical to the policy process, particularly as frames of reference that buttress existing policies or challenge those policies with alternative problem definitions, theories of action and programmatic strategies.  </P> <P> In education, one can easily think of ideas that have shaped policy - fiscal neutrality, school choice, standards-based reform.  However, many empirical researchers, including ones doing policy-relevant research, are uncomfortable thinking about applications of their work in terms of ideas because they see them as essentially ideological and not evidence-based.  Certainly, many of the ideas animating policy debates lack a firm research base, but if we are truly interested in understanding the politics of knowledge utilization, we need to focus on the interaction of ideas, institutions, and interests that produce policy and the role of knowledge in that process.  And this brings me to my final suggestion.  </P> <P> In the 1970s and 1980s, scholars such as Carol Wise and Nathan Kaplan systematically studied the use of social science research in policy decisions.  Now, that strand of research has all but disappeared as researchers and funders have become less optimistic about identifying the enabling conditions for research utilization.  However, if we are serious about understanding the institutional and political conditions under which research is more or less useful to policy makers, we need to update those earlier studies of research utilization and to study the effects of our own work as systematically and with as much rigor as we study the effects of schooling.  And I think the papers presented on this panel and what you will be hearing the rest of the day represent a fruitful start to that project.</P> <P>   Frederick M. Hess:  Thank you, Lorraine.  Reid, hit the button. </P> <P>Reid Lyon:  I did not write a paper.  I did not put anything in writing; I should have learned that with e-mails.  One of the interesting things about leaving Washington is you really do understand it is 30 square miles surrounded by reality.  I m going to try to provide some concrete language to follow Mike Feuer s admonition to be concrete and very accurate about how a great deal of what both Jim and Richard have said -- and to be concrete -- and let me thank Rick for asking me to be here.  Thank you very much.  And I hope this does not sound like I m focusing on myself but I m not sure if people really understood how Reading First came about.  In fact, I m sure they don t. </P> <P>And so I m going to talk you through it in 10 minutes, basically, and try to weave into that discussion the concepts that have been presented.  And this is a question that has been baffling everybody; everybody has talked about so far:  Why is it tough to use trustworthy scientific evidence to inform and guide educational policy and practice?  </P> <P>Now one of the points I want to make is that even the title of this conference focuses on policy rather than practice.  And what we found in developing the legislation for Reading First and NCLB as well is we did not get traction until we started talking about instruction.  I ll try to clarify that.  There are a few reasons -- and everybody has covered these already.   Ellen Lagemann has written about this for decades extremely well; I will not belabor this.  But clearly one of the things that we have had difficulty with is infusing what we know about how kids learn, particularly in reading, into the fabric of schools and instruction and at colleges of education, and anybody that needs to know this information.  The anti-scientific culture probably is not education s fault; it is more the fault of people like me, where we did not present research findings in a relevant, clear, accessible way.  Some of the times, my eyes glaze over when I try to read things that I think are important to kids and instruction and reading.  </P> <P>So, the anti-scientific culture which everybody has alluded to this morning is one where our practices have been driven by tradition and philosophy and untested beliefs.  And those do gain traction fairly quickly because, as has been pointed out, they resonate quickly with those people on the line in very, very complex jobs called teachers.  And we talk about teacher- bashing; there in no way is any teacher-bashing inherent in what they have to do.  They make as many decisions - and people may not know this - as an air traffic controller per day, an amazingly tough job.  But what they do in relationship to their informational base is problematic, and I ll cover that shortly.  </P> <P>Complexity is tough to sell and Jim and Richard talked about how things have come about.  I liked Richard s particular framework because, as you know, in science we are always talking about how you can screw research up by picking the wrong method or design; you can screw research up by making the wrong interpretation.  But the big screw up is asking the wrong question. </P> <P>And Richard s paper clearly unpacks that we have asked the wrong question in terms of why do we have so many out-of-field teachers, and it is not for the conventional wisdom.  It is because, as has been pointed out, administrators have not received sufficient background or knowledge to be able to make more efficient decisions and appropriate decisions.  But, indeed, politics trumps science and that has always been the case.  There has no been no market for science at all and I think that relates back to our inability to write clearly about what we know; and it also relates clearly to not doing research in a very focused way. </P> <P>A long time ago, given what I just talked about, we asked three questions and stayed with them for 30 years:  How do kids learn to read?  What goes wrong when they do not?  And what you do about it?  That is it.  Thirty years of just asking those same questions over and over and over again and building into the answering of those questions research sites that would look at them from a multidisciplinary perspective:  education, psychology, cognitive neuroscience, pediatrics, radiology, every site -- all asking the same question so we get to the fundamental replications.  </P> <P>And, indeed, out of this research became what would be the nexus with policy.  And what we found over those thirty years was not, in particular relationship, to Jim s paper -- it is in direct relationship to this.  I do not understand why we cannot get our hands around complexity.  I do not understand, no matter how many times we say that it is not what research method we used.  It is not qualitative or quantitative; it is how you apply the different methodologies alone or in combination to the right question.  But notice the polarization.  Jim brought up phonics, whole language.  Where in the world does that come from other than ideology?  </P> <P>In the research program that I just pointed out, the question was not Is phonics or whole language more beneficial for kids?   It was, For which kids are which instructional approaches or combinations therein most beneficial at which stages of development within which settings by which teachers? It is hard for the press to get their hand around that.  That was the question.  And when you ask the question that way, unfortunately, you get an equally complex answer.  If you want to understand what you read you have to be able to do it quickly but that takes all kinds of things to do.  If you want to be able to understand what you read it is even more difficult.  You have to know all of this.  If you want to be able to comprehend or understand what you read, it takes all kinds of other things and if you want to comprehend or understand what you read, it takes all kinds of other things.  </P> <P>Now, I provided Congress, particularly Bill Goodling, all of these data in 1997, and it was the first time he had seen anything like this.  And he said, Do you mean to tell me that we have this gap between where our kids are and what the science shows?  Do you actually mean to tell me we know how to do that better?   And I said, Yes sir, we certainly do.   Because we had been getting the intervention data in now for eight years when I talked with them.  There came the impetus for many broad-reaching consensus panels for the Reading Excellence Act.  That is the first time you have ever seen scientifically-based reading research directly from this, a conversation between a policymaker who is worried about 60 percent of poor kids not able to read.  He calls NIH and he says, Can you come brief me.  I hear you are doing some work.   He was a high school principal.  His wife was an early childhood educator.  </P> <P>For some reason Bill Goodling, then Mike Castle, then Ted Kennedy - Kennedy all the time, actually, is always interested in this - became engaged in this process.  I m not sure how all of our policy discussions covered this, but that is what happened.  We put Reading First into -- the Reading Excellence Act into place and we defined scientifically-based reading research for the time; but it went nowhere.  It went nowhere because we did not ask people to meet criteria that defined scientifically-based reading research, which is all of this.  More clearly, for a program to receive federal funding, that program would have to have been found to be effective in improving reading capabilities.  </P> <P>In the Reading Excellence Act that was watered-down to based-on reading research because there where not many reading program that where available.  Likewise, in Reading First, I wrote the same language which -- federal money can only be available for those reading programs that have been found to be effective for well-defined children under specific conditions, including teacher, classroom, age, developmental status.  And you will see that in the IG report if you ever read that about Reading First; that was not accepted.  It went to a much balder, blander kind of criteria, based-on.  There is lobbying pressure that did that but there are also practical pressures that did that.  </P> <P>That is how a policy was driven by research.  It got away from polarizations that Jim is talking about - phonics, whole language - and it was based upon research which was, indeed, not polarized.  It was for which question or which research approach is most beneficial to answer that question.  </P> <P>I ll just close with this:  What do we need to do?  I think - and these are just suggestions - that we do not have a common language; that is, our researchers continue to polarize dramatically.  And in my 1998 testimony this is what I said: Debates persist about the merits of conducting quantitative research versus qualitative research.  In reality, the scientific methods of both research designs are equal to the extent that they are applied appropriately to the questions under study.   That was as far back as 1998 and it was repeated every year after that to policy makers.  If we do not have researchers that are trained to do that, it is not about the quality of research, as Mike was pointing out some educational research quality is tremendous.  </P> <P>It is just that we do not train our students who will use it and study education when to apply the appropriate methods.  I mean, that is a simplistic kind of solution.  Likewise, teachers are frequently presented a one-size-fits-all.  Jim brought this up - a philosophy that emphasizes an either-or language or phonics orientation.  This parochial-type of preparation places many children at risk.  In other words, it is way too simplistic; it does not take into account where you deliver instruction.  It does not take into account that no one reading program is equally beneficial for all kids.  It does not take into account that teachers have to know what they are doing in terms of that very large slide so that they can modify instruction when kids do not get what it is they have to learn to read appropriately.  That has not come through, unfortunately, in consensus panels.  It has to come through in some other way, and in a lot of ways that has to be more of a bold statement.  There is a lack of urgency in what we are doing.  </P> <P>So, we have to create the urgency that says we have a problem:  What we know is not being applied; the gap is not closing.  And that has to be recognized and that was what, in fact, drove Reading First and No Child Left Behind - a sense of urgency, science reaching some momentum, and a policy-setting that allowed it to happen.  Thanks.</P> <P>Frederick M. Hess:  Okay, thank you, Reid.  With that, Jimmy, Richard points to add?  Clarifications you want to make? </P> <P>James Kim:  I ll say a couple of thoughts.  I think one of the themes that we have heard throughout this morning is that why cannot education be more like medicine.  And Paul Starr, (he is a sociologist) wrote a Pulitzer Prize-winning book about the social transformation of American medicine.  And he pointed out that professional knowledge rests on three claims.  </P> <P>One claim is that the knowledge is consensually validated by one s peers.  It is not the individual who decides what is excellent; it is your peers.  The second is that this consensually-validated knowledge rests on some sort of rational or empirical scientific grounds.  And the third is that the profession is oriented toward a substantive value like health and the well-being of children.  </P> <P>I do not think anyone disputes that education embraces the third quality that most teachers, principals -- we want children to read well, to do Math, to have their life chances improved.  I think the second one about knowledge resting on rational grounds -- NCLB has certainly done a lot and we just heard a presentation about the importance of scientific evidence and informing instruction and practice.  But I think education and teaching is a partial profession because we miss that first one.  There is no mechanism right now to have really excellent practitioners, really excellent teachers, form a consensus about this is what excellent practice looks like.  </P> <P>And so my recommendation at the end of my paper of, maybe, having consensus panels that look a little bit more like the United Kingdom s Literacy Panel.  It is something, I think, we do not try in America but I think it is something worth considering, mainly, because we have to realize that sharing power over the governance of curricula is important because it gives practitioners an opportunity, I think, to be invested in the policy-making process and, ultimately, we are asking them to carry out the reforms that Reid and others have talked about.  So, that is one thing that I wanted to say about the importance of teachers in the policy-making process.</P> <P>Frederick M. Hess:  Richard? </P> <P>Richard Ingersoll:  Yes, I want to just react to, I thought, a great insight Lorraine added about why certain interpretations get dominance over a particular -- you know, why some data arise, and I found myself wondering this so many times over the last decade-and-a-half.  We present these data showing that X-percentage of teachers in field Y are not qualified.  And not always, but almost always, the immediate interpretation on the part of consumers, whoever they might be, would always be, Gosh, these teachers lack something.  These teachers have a deficit.  These teachers are at fault.   That s very quick jump.  And I always puzzled why.  </P> <P>It always seemed very straightforward to me.  There are two issues.  There is the preparation you get before you are on the job and there are, certainly, problems with that.  And then there is how you are utilized once on the job.  It sort of seemed like two things, and these data are the second thing.  The first thing is certainly a problem, and I always wondered why that distinction was so hard to get across.  </P> <P>It has been such an up-hill fight and I think Lorraine has provided a pretty straightforward answer.  One answer suggests it is the problem with preparation in schools of education.  And that is an ongoing - that is not a new issue.  Indeed, I think the teacher quality wars were already well into battle when these data came along, and so you already had positions and you already had polarization.  It was almost like new ammunition arrived that could easily be brought in to different positions in sort of the larger issue of what is a qualified teacher and do we need schools of that and do we need certification, et cetera.  </P> <P>And, on the other hand, the interpretation I m offering is far more murky.  It does get into the internal management of schools and that is a less easy road to go down.  Certainly, in fact, with the cases of the federal government and NCLB, internal management of schools has traditionally been off-limit to legislation.  You cannot really go there.  So, I can see how it is much more attractive to think that the problems are with education schools rather than in how our high schools are set up and organized and run.  I mean the latter is a very large problem to tackle and who wants to tackle it?</P> <P>Frederick M. Hess:  Okay, let s go ahead and open it up back here.  Actually, Juliet will get the mike over to you.</P> <P>Paul Kimmelman:  Thank you, Paul Kimmelman, Learning Point Associates.  I ll direct my question to Reid but I think it can pass along to all of the panelists.  First, a quick context- setting point; I think the medical model makes considerable sense for those of us in education but it, too, has not been without its faults and problems with its research implementation as well.  But, based on your experience now, Reid, after watching NCLB become law, what are some suggestions you might make to the House and Senate Committees re-authorizing the law that would enable them to implement a more successful evidence-based concept for school reform initiatives?</P> <P>Reid Lyon:  One of the things that we have obviously seen is the capacity to take that legislation and implement it for a wide variety of reasons.  We have to spend much more time.  A number of people have pointed out the need to understand the conditions under which broad change mechanisms can be implemented and sustained.  Clearly, that is one.  </P> <P>A second issue with Reading First, in particular, goes to this point and it has been a tough one.  We knew that when we watered the language down in Reading First we were going to have problems; that is, when we pulled off the fact that federal money had to be spent on that which had been shown to be effective under which conditions, pulled off that and put based on, a couple of us working on that looked at each other and said, This is going to be a nightmare.   You have money involved and you have a very nebulous criterion.  And watch - in six months, everybody will be evidence-based.  Everybody will have SBRR.  And how are you going to manage that?  </P> <P>So, number one, in Reading First I doubt if it will get through but that is the recommendation I have made back to the Committee, actually the same language I wrote for the Reading Excellence Act in 1997 and for Reading First in 1998.  What you find with Reading First and NCLB is the need for dramatic technical assistance as you move these programs out.  People do not know what SBRR is.  They really do not have any idea in many places.  It is not that they are bad people; it is we have not given them that information.  There are many other adjustments to NCLB that are a little goofy; I mean, the NCLB is goofy.  Highly qualified teachers really need to be addressed.  Special Ed issues need to be addressed.  There are a number of shortcomings in that law that all have to be addressed.  Most of them are extremely well-articulated by the man to my left.  Hope that helps.</P> <P>Lorraine McDonnell:  I just thought of this and it may be off the wall, but I was thinking about in the past when ESEA was re-authorized.  Congress would set up a study that belonged to Congress.  I know Bea Berman [phonetic] is in the audience and she is probably cringing when I m saying this.  But those were interesting studies and I was thinking about how they may make more sense now when you have many voices coming in because they belong to Congress.  And Congress could look at how to fix things incrementally, which certainly seems to be the case now because you want some consistency over time.  But I m just wondering -- this is part of, I think, the diminution of belief in evaluation and the uses of research that may have happened in Congress.  But maybe getting back to that kind of thing where those studies really do belong to Congress might be important for re-authorization in the future.</P> <P>Chris Mazzeo:  I m all the way in the back in here.  Chris Mazzeo, independent consultant based in New York.  I actually want to ask a question that was based on it was almost an offhand comment that Lorraine made in her comments and it is thinking about, sort of, if you look at the subtitle of the heading there, Why Education Research Does (or Does Not) Influence Policy, I think so much of the discussion is not so much here but when people have these kinds of conversations really rely on issues of faith and anecdote.  </P> <P>And, I guess, I want to ask the panelists, and I would actually like everyone to think about this in the future -- if the other panelists after lunch -- to think about how do we know?  I mean, I think that one of the most vexing questions here is that we do not necessarily know very well, from a rigorous way, what the very answer to that subtitle is.  Under what conditions does research influence policy or practice?  And what are the indicators, what are the metrics?  I have a few projects on my own now in my own work, and I think about this issue.  I would not even know how to answer that question in a lot of ways.  </P> <P>And certainly Carol Wise s work and other work going way back gives us some indication.  But I think it is something we need to think about, to not presume that it never does, to not presume that it does in certain cases but to really actually get it in detail.  How, under what conditions, did research influence issues of literacy and reading?  How did it influence teacher quality debates in policies in states, in communities?  </P> <P>Frederick M. Hess:  Yes, I think you just found it right there.  So, Richard, Jimmy?  How do we know?  How does one think about this?  What are useful ways to construct the question?  What are the metrics worth looking at?  Lorraine?</P> <P>Lorraine McDonnell:  I ll just give you time this way.  I think that is the question that we need to look at and we need to put the earlier research.  For example, Carol s notion about the enlightenment function, that it happens over time; I think we now also have - and I m speaking as a political scientist - we have the framework.  I was thinking about Jimmy s paper where he said he wanted to look at the institutional mechanisms.  I mean, we can take that research and we can put it now in a framework where we understand a lot of this is going through ideas.  </P> <P>We have a lot of research on the politics of ideas.  We have research on the role of institutions and interest.  And I think we can slot research utilization into those existing theories in a way that was not done in the 70s and 80s.  But I agree; I think we need to have a rigorous kind of research agenda on knowledge utilization.</P> <P>Frederick M. Hess:  Reid?</P> <P>Reid Lyon:  I think the implementation or the use of research in policy requires, first, the articulation of an urgent problem; maybe I m saying the obvious.  We had from the NIH, from the National Academy, NRC, from Dick Anderson s group -- we had many, many extraordinary consensus reports on reading, for example, that would have driven policy very nicely but they did not.  And they did not because there was no urgency; people did not understand the gap until it became publicized.  There was actually then a market for it.  It was not until, I think, President Clinton in 98 said in his speech that he could not understand how we had 40-something percent of our nation s fourth graders failing to learn to read.  </P> <P>So, in a sense, it is a backward set of circumstances it seems to me.  Again, Reading First was not born out of our reports; it was not born out of the consensus panels.  It was born out of a legislator finally getting it, that a gap exists that does not need to be there, and that gap was a political value to that individual.  This may sound way too simplistic but the current legislation is in place because of those mechanisms;  not because Congress sat back and the framers of this looked at a tremendous body of research on how you translate policy into action at all.</P> <P>Frederick M. Hess:  Jimmy?</P> <P>James Kim:  I think we have to create incentives for the folks that we are asking to use this knowledge to want to use it.  And I think that Reid has articulated kind of his story of how the Reading First legislation got enacted.  I think an equally valuable model is something like the National Research Council s Strategic Education Research Partnership where, basically, the goal is to have the superintendents meet with the professors to talk about issues of practice that they want some answers to.  And, I know, in being involved in that initiative, what is very important about that is it is not the research community that is saying, Look, here are the questions you should look out because of theoretical reasons or because it is just a debate or we do not really know the answers to, but, What is the question that is important to you and what is the answer that you need to improve practice in student outcomes?   </P> <P>And I think if we do that and we have the district superintendent and the principals and the school teachers asking those questions and the researchers help frame the design that there is an incentive for whatever research finding to emerge from the study to be taken seriously, to be used and, hopefully, implemented in the classroom.</P> <P>Reid Lyon:  Yes, I just want to mention, I think, that is the same thing that happened at the federal level.  There was a market for the research.</P> <P>Richard Ingersoll:  Well, I guess the question is: How do we know whether this research has actually impacted practice on the ground?  Was that the question?  And it goes back to Michael s earlier thing; I think, it was the scientists propose and the politicians depose or something like that.  And in a sense --</P> <P>Frederick M. Hess:  Depose -- seems accurate does it not?</P> <P>Richard Ingersoll:  Dispose -- sorry about that.  From a researcher s viewpoint you do not know; it is just a big black hole in a sense.  I mean you do this research; you it put out there and you cannot -- you neither can control or you really know who makes what of it in what ways when and how, in a sense.  And I guess you could try to do research.  I mean, another thing I do a lot on is how many beginning teachers get induction, support, mentoring, and these types of things?  And does it work?  </P> <P>And so, I can look at the national data and I can track over time and see that, indeed, there is a trend that more and more beginning teachers get these kinds of support and I can also see what they are.  Now, is that because of research showing that it works?  I do not know.  And I do not know how I would know.  I guess you could try to figure that out but in a sense it is sort of beyond us.  We put it out there and then it diffuses or does not.  That is not a satisfactory answer and I guess there could be.  I mean, Lorraine is suggesting concrete research that could actually look at diffusion patterns or something.</P> <P>Frederick M. Hess:  Okay.  Yes, ma am?</P> <P>Donna Dickman:  My name is Donna Dickman.  I am a classroom teacher, 16 years.  I m actually now a staff development teacher in a suburban county of DC.  And, I guess, what I like to ask is what you actually were speaking to which was in terms of the I hear we are hearing a lot of talk here about whether or not this actually comes into practice and we are here almost dumbfounded because our worry is how much it has influenced practice and is anybody paying any attention to that.  </P> <P>So I want to assure you the things you are talking about are influencing practice.  Our fear is in not always in very positive ways and I just wonder if enough attention is being paid to that.  A couple of [indiscernible] of some of what has been coming in the last six, seven years is that -- and I have traveled the country for the last five years in many, many schools.  And I can tell you that what is happening is science is not being taught in school anymore in the elementary schools; it does not exist in many schools anymore.  There is very little writing being taught in schools anymore if it does not fit into a box that can fit into a test.  Higher-level critical thinking is not being taught in schools anymore.  And real-world application is not occurring in schools because we have tremendous time constraints and a lot of the programs that have now been imposed on the schools are not fitting within the time frame.</P> <P>Frederick M. Hess:  Okay.  So the question that comes out--</P> <P>Donna Dickman:  So, the question is -- well, and it is just I know this is not something that is going to come out today on the panel so I would like there to be some teacher representation today  -- is what is happening in terms of research in looking, looking at how it is being diffused, what those costs are, and what those next steps are going to be?</P> <P>Frederick M. Hess:  Okay?  </P> <P>Reid Lyon:  Written into the legislation for NCLB and Reading First are some very massive evaluations; one is $15 million a year just to look at Reading First.  Whether or not that gets implemented -- well, who knows?  The GAO has looked at some diffusion and dissemination patterns and the effects on practice and teacher response; the OMB has done the same thing with Reading First and so on.  Those look pretty positive but those are not, in any way, results of effectiveness.  But it is asking enormous numbers of school leaders and teachers whether or not they are using the information, how efficient it is, what they need to be able to do.  You might want to look at the GAO report and the OMB report.</P> <P>Frederick M. Hess:  Jimmy, in your paper you alluded to this tension, at least in the 80s early 90s, reading disputes in which classroom teachers and, you suggested, professional associations tend to embrace an approach which the policy makers, particularly the policy makers in DC today, have tried to challenge and have tried to change, obviously.  How does it play out in a case where there is a concern that the classroom is being altered in ways that teachers and professionals are uncomfortable with?  Does that change the dynamics of your analysis?  What does one make of this?</P> <P>James Kim:  Let me make sure I understand your question.  You are saying how does so if research, if the science, is good and then is implemented in the classrooms and there are unintended consequences, or intended consequences, how does it change the way we should think about the research?  Is that what you are asking?</P> <P>Frederick M. Hess:  Well, that, or what does it mean in terms of this professional autonomy question that you raised,  the fact that teachers obviously have reasons they are going to prefer certain kinds of practice or research as better disseminated?</P> <P>James Kim:  Well, I think this exchange we had is really useful to think about because as researchers we have a luxury of thinking about really narrow specific questions.  I mean, I think Reid did a great job saying, Look Nietzsche [phonetic] for 30 years was trying to answer this question, Which interventions for which kids at which stage of development under which teachers?   We can ask these questions.  We can study for 30 years.  If you are a classroom teacher, teaching reading is a tiny portion of your job and I think that is what this classroom teacher is trying to communicate to us:  It is just one small part of her job and there are many other subjects that you have to teach.  You have to teach math, you have to teach science and social studies.  </P> <P>So, I think one of the things that we always have to think about is this question of external validity.  We can do a lot of great experiments; we can accumulate a lot of evidence.  But then when it gets implemented in classroom settings, let s say, where there is a high-stakes accountability system in place, what does that do for the measurement of fluency, let s say?  </P> <P>This is very important outcome that has to be measured under Reading First.  All of a sudden, it is about not just helping children read fluently; it is just reading fast.  And so that gets caught up into the evaluation system and so a lot of these measures that were used very well in the research setting get corrupted.  So, I think that is part of -- one of the things that we have to pay attention to; it is just the external validity of a lot of the research that is being translated from these very elegant experiments to the classroom settings when we have these kinds of accountability systems in place.  And I think a lot of the push-back is the result of that.</P> <P>Frederick M. Hess:  Okay.  We will take one more quick question back here.</P> <P>Scott Joftus:  Hi, Scott Joftus with Cross and Joftus. General comment, and then maybe a suggestion, James, for your recommendations.  It seems to me that the idea of research to policy has been constructed in kind of a traditional sense in that it is external researchers pushing research down into the classroom.  And I would urge us to think about the power of getting teachers engaged in the actual process of studying their own practice in a way that improves what they are doing.  And, James, that might be something that you would consider for one of your recommendations.</P> <P>Frederick M. Hess:  Well, in fact, I mean I think what we have heard here, what Reid alluded to was this feedback loop in which sometimes we talk about it as researchers or sitting on the edge and kind of firing things into the game and hoping somebody picks them up.  But there are, of course, these back-and-forths in which researchers are approached by policy makers or encouraged to tackle certain things which then come back at a later date, a year later or a decade later and have an impact on policy.  </P> <P>Any closing words, Richard, Lorraine?  If not, why do we not let that be the last word.  [Applause]  You will find lunch set up right outside in the foyer and we will reconvene at about quarter of.  Thank you.</P> <P> </P> <P> </P> <P>[Start of Panel 3:  How Research is Used NCLB and School Choice] </P> <P> </P> <P>Frederick M. Hess:  If folks would go ahead and take their seats, we will get ready to resume.</P> <P>Again, let me just remind you all for those who joined us for the afternoon shift that the papers are available out in the lobby, got hard copies of all the papers.  We also have CDs with the full set of papers and you can find all the papers available online.  At the web address we have up here, <A target=_blank href="http://www.aei.org/event1455" target=_blank>www.aei.org/event1455</A>.  All the papers are up there and will remain available up there, I believe along with the PowerPoints that are shown today.  Again, let me just remind everybody that the questions we are focused on with these issues are not necessarily of the quality of the education research in any narrow or prescriptive sense, but these larger questions of the relationships between researches and policymakers and more broadly why research does or does not influence education policy.</P> <P>We have seated now the third panel on the day.  This panel is going to look at the question of impact of research in the case of school choice and No Child Left Behind.  Saying my ground rules as this morning, I will ask when we go to Q&A again that people please resist the temptation to orate from the floor and please be kind enough to identify yourselves by name and affiliation when asking the question.</P> <P>Seated up here in the order that they will be speaking, first, we have Andy Rotherham.  Andy is co-founder and co-director of Education Sector and a senior fellow at the Progressive Policy Institute.  He is also author of the blog Eduwonk.com.  During the Clinton administration, Andy served at the White House as special assistant to the president for domestic policy.  Presenting second will be Mike Petrilli.  Mike is vice president for national programs and policy at the Thomas B. Fordham Foundation.  He served as a George W. Bush administration appointee in the US Department of Education where he helped coordinate No Child Left Behind s public school choice and supplemental service provisions and where he oversaw discretionary grant programs for charter schools, alternative teacher certification, and high school reform.  He is also a co-author of the aptly titled No Child Left Behind Primer.  </P> <P>Speaking third is Roberto Rodriguez.  Roberto is a senior education adviser to Senator Edward M. Kennedy on the Health, Education, Labor, and Pensions Committee.  Before working on the Senate Committee, Roberto was a senior education specialist at the National Council of La Raza, where he engaged in applied research and policy analysis of federal and state education reform issues.  And then, on my right here is Dave Driscoll.  Dave is the commissioner of education for the Commonwealth of Massachusetts.  He was appointed Massachusetts deputy commissioner of education in 1993.  As deputy commissioner, he was the principal investigator for the National Science Foundation s mathematics and science program in Massachusetts, and was instrumental in gaining the NSF s approval of a second five-year round of funding for this initiative.  Dave is wrapping up how many years it has been?</P> <P>David Driscoll:  Forty-three.</P> <P>Frederick M. Hess:  Forty-three.  He will be retiring actually as commissioner this August and has kindly agreed to come down and get himself a little bit of trouble on his way out of office.  With that, Andy, would you please start us off?</P> <P>Andrew Rotherham:  Thank you, Rick.  And I want to echo.  I just want to thank you for hosting this conference.  It is an interesting set of issues and a terrific set of papers.  I cannot linger too much with the preliminaries and the niceties because they are watching me closely on time.  But I want to make sure to say that.</P> <P>My project was to look at school choice research, whether or not it has had an impact and how it is played out.  And it really was a challenging assignment because I really thought sort of in the spirit of one of the other hats I wear that in some ways the answer of the question is a blog post.  You can knock it out in 500 words.  School choice is very ideological.  I'm not saying anything that anybody in the room does not already now.  People behave in an ideological manner and evidence changes few minds on either side regardless of how it comes down.  </P> <P>And so, what I did because I was not asked to write a blog post is that I broke it out into sort of the two questions, which I think are frequently conflated.  The two choices in this debate that are frequently conflated and those are, first of all, what range of educational choices should government finance and then second, what are the effects stemming from the various choices policymakers can make regarding educational choice.  And what I argue is that one of those questions is fundamentally an ideological question.  It is a question that reasonable people can disagree and they are legitimate competing views and that we often inappropriately try to use research to try answer and use as a proxy for those larger positions in inappropriate ways.</P> <P>The second one where the effects stemming from the various choices policymakers can make is exactly the kind of question where good social research can really inform the conversation, shed light, add value, and help policymakers think through the kind of choices that they make.  So, I look at this through the lens of three separate issues.  The politics, that just surround the larger choice issue, the actual research debates, and then third the role of the media.  I know some people this morning bristled at the language of translators and so forth.  </P> <P>But the fact of the matter is, the media is there.  They play a key translating role here, not only in terms of -- we really have to look now further, for example, the AFT/New York Times that we are talking about sort of the effect the media can have and kind of firestorms they can set off.  But also, when you go outside of rooms like this, you have people who tuned into issues only episodically and so they look to the media for various cues.  What they should think about issues and I think all of us probably do that on some other issues that we do not follow so closely.  For example, the New York Times really shapes how you think about an issue that you do not follow so much.  So, I really delve into the media some.</P> <P>The politics, pretty straightforward.  First of all, they are very ideological.  There is a passage that I reproduce in the paper by the economist, Herbert Gentis, and I'm going to read itbecause it is worth reading.  I think it is very illustrative of the debate.  In a foreword to a recent book by the late Eric Rofes and Lisa Stulberg about sort of the [indiscernible] promise to charter schools.  So, there is a progressive case for school choice.  Gintis wrote in the foreword, When I wrote the political economy as a school choice for Teachers College Record some 10 years ago, my progressive friends thought I had lost my sense of reason.  Everybody knew that school choice was a conservative plot to finance the private education of the well-to-do, to bleed the public schools of needed revenue and to add one more roadblock against the struggle for social equality.  Indeed, when I started writing about education in the 1970s, I shared this view.  Not that I really thought about the matter, I just knew that if Milton Friedman was for it and the teacher s union were against it, I should be against it, too.   And that is a powerful and brave admission and I think it shows how ideological the debate is.</P> <P>Second, it is organizational.  You have big organizations vested interest in the current system, positions of privilege, and so forth.  And so, there are huge organizational politics.  And with that creates a research trajectory particularly around the voucher issue and you have sort of a group of folks more in the policy margin - this is sort of an old story in politics - who are producing the research.  The trajectory tends to be that research gets produced about school choice showing some sort of effect and then established organizations debunk that research and so forth.  And that is a sort of very common trajectory.  It is an old story.  We get these policy actors who are trying to get their issue on the agenda because, well, passive school choice, like residential choice and things like that, are a key feature of our educational landscape today.  Active choice is still a more marginal reform.</P> <P>And then, third, framing in politics matter more than evidence.  And the story I tell about that is really the Terry Moe book when it came out from Brookings.  That book, at one level, it is sort of a dry analysis of bureaucratic behavior.  It was an important book.  It was good thesis scholarship.  But what was really important about it was actually none of these things.  It was that it was published by Brookings, and it was perceived as a school choice book and that started to change sort of, again, to the Gintis quote, how people started to think about this issue.  And the framing in the early 90s that continues to this day of school choice is not a free market issue, not sort of a Milton Friedman issue, but it is a poor kid issue.  What you see, especially associated with kids of color, communities of color, the work that pHoward Fuller has done, reframed the debate.  </P> <P>And frankly, if you sort of look at the sweep of these things in American politics, sort of the expansion of the liberal school choice programs over the last 15 years, it is really quite noteworthy.  So, it is not political framing that has really had the impact.  So, I got ahead of my slides.  So, anyway, that is why it is told to politics.</P> <P>The research.  A couple of big trends.  First of all, the research has moved from being descriptive to inferential.  The early work around the school choice whether there is a stuff of Milton Friedman s work or stuff people like Chris Jenks, Jack Jennings, Steve Sugarman were doing, it was descriptive work and theoretical work and that was in part because there was no data.  Second big debate, more recently, I should say, more recently the work has become much more inferential and that is impart because of the standards movement with the kind of data we have now.  It is much -- we are able to do much more sophisticated research on these questions than you were and we have more school choice programs to evaluate in the first place, after Milwaukee and Cleveland, now there are just more places where the researchers can go.</P> <P>Second, I think what has characterized the research big debates -- Jeff talked about a couple this morning, the big Witte-Peterson debate and so forth -- but, actually, in the big scheme of things these had relatively small effects and both sides are sort of ascribing very powerful proxies to this effect as to whether or not choice is a good idea.  But, actually, in the big scheme of things, these are not particularly powerful findings one way or the other.</P> <P>Third - and this is something I get into in the paper - it improved the research.  The interesting thing about the Witte-Peterson dispute today is first of all, they are working together on some other projects collegially today and, second, they both say it improved the work and that some of these disputes were classic social science with a back and forth that improved the work, but it really coarsened the debate.  </P> <P>And one of the things that I found really striking, particularly as the stuff was translated by the media, is just how personal the debates become.  And particularly in the case of Paul Peterson, a lot of the articles you read about the dispute, they were not actually about the dispute.  They are about Paul Peterson and whether or not he was trustworthy and so forth and it became intensely personal.  And so, on the social science level it improved the research and really coarsened the debate.  And then, I look at some key challenges - data availability, peer review, and the incentives for researchers, and, particularly on the incentives point, researchers are really in a bind here.  The foundations are increasing.  They want you to have an impact.  They want you to get out there and be in the mix.  But the minute you sort of step out of being a researcher and you start writing, for example, papers or op-eds for the Wall Street Journal, say, you step into the political debate with everything that goes along with that and so the incentives are a little bit tricky.</P> <P>And finally, media.  I talked about it in terms type 1 and type 2 errors in research.  Journalism and social science are, to some extent, at odds.  Both fields prize accuracy and reporters certainly and papers and magazine and so forth have really redundant processes built in to try to make sure that things are accurate, but there are some really key differences.  </P> <P>First of all, journalistic stories are time bound.  They are writing a story about what is happening at a point in time or relatively recent to that whereas (and Jeff, I think, did a good job talking about it this morning) social science is predicated on the steady accretion of knowledge and there are lots of answers where it depends.   But in journalism you need some sort of a punch line generally for a story and they are time bound.  </P> <P>Second, these are really complicated studies and the answer is often it depends.   Newspapers do not like to publish stories, particularly on their front-page, of studies with a conflicting and inclusive data mix.  They want stories. This is what you saw, for example, in the 2004 New York Times story that had a punch line.  It tells you something.  It is binary and that is a very awkward frame.  </P> <P>And then, third, it is just a contentious advocacy debate.  There really are very few disinterested observers around and for the reporters there are really confusing cues. One editor at a major daily told me, as I was talking to her about this, that they look at cues. For example, if the government funded a study then they are more likely to write about it.  But that actually, there can be plenty government-funded studies versus some of the stuff that is coming out, NCES.  It is of high quality, it is interesting, but you cannot make causal claims based on it, yet they are using that as a cue because there is so much conflicting research and reporters get very confused.</P> <P>So, I'm a policy guy so I can t help but make, like Jimmy did, some recommendations and so forth and a conclusion.  My primary conclusion is research has actually mattered relatively little to the trajectory of this debate.  It has informed a few things and so forth, but the trajectory of this debate is that big organizational and ideological politics are shaping it.  </P> <P>Second, the research can inform discrete aspects of the choice question, help policymakers.  But in a liberal society, it simply cannot answer the question of what range of choices and education you want government to finance.  Where do you want to invest the basic locus of decision making with the parent and state and the relative relations there in terms of kids?  Research is not going to be able to tell us that and we are going to continue to be frustrated as we try to use these different findings as a proxy for answering what is fundamentally a very complicated question.  </P> <P>Third, by way of recommendations, more training and support for the media and the work people like Richard Colvin are doing.  We cannot expect reporters to be experts on these things, but we can expect sort of a higher level of stories we put in greater context.  We can expect more sophisticated consumption in the research than a lot of back and forth that you see now. </P> <P>And then, finally, for researchers, more attention to research quality and data issues. There are a couple of points I get into in the paper.  One is the data availability question.  People say everybody should release their data.  It is bad that Paul Peterson does not release his data.  John Witte might not release his data either and he makes a very compelling point that the investment in cleaning data and doing this kind of analysis is so substantial that you create tremendous disincentives if you do not let people have exclusive access to the research for a period of time.  Witte s point is he did all the this work on Milwaukee and he didn t even published the first journal article about the effects of the Milwaukee voucher program because under the open records law, somebody else was able to come in and do that.  And so, we need to start to think creatively about ways to maintain exclusive access, but also address the issues of transparency and peer review.  </P> <P>And then, finally, we are starting to see some efforts, like the National Charter School Research Project at the University of Washington, to lay out rules of the road and that is really essential in terms of starting out.  That will help media in terms of taking cues and having some established standards about different kinds of research, where it is applicable, for what it can and cannot tell you, and to start to enrich the conservation a bit more that way.  So, thank you very much.  The time-keeper is giving me an evil eye.  And so with that I look forward to your questions.</P> <P>Frederick M. Hess:  I told them to give you an evil eye whether or not you go long.  Hey, thank you, Andy.  Mike?</P> <P>Michael J. Petrilli:  Okay, well, hello everybody.  I was thinking on the way over here today that this is now my fourth time speaking from this podium on No Child Left Behind in the last two or three years.  I think the last time was, of course, to plug the wonderful primer on No Child Left Behind available on amazon.com or for your educational school courses next fall.  And some would argue that each time I have spoken on No Child Left Behind, I have taken a slightly different position.  Now, some have argued that that is contradictory.  I would use the word nuance.   I think Mr. Eduwonk over here talks about the road to [indiscernible], which -- could you explain that?  What does that mean?  What is that supposed to mean?</P> <P>Andrew Rotherham:  Well, it is a play on words.  Your boss, your friend, your colleague, at Fordham, Rod Paige and you seemed you had some sort a conversion at some point on No Child Left Behind.</P> <P>Michael J. Petrilli:  I see.  That is very nice.  So, I'm glad to be back and to talk about No Child Left Behind.</P> <P>Before I start, I really need to give credit to my collaborator, Paul Manna, at the College of William Mary, who was in China right now, which is why he is not here giving this paper.  Anything that you find scholarly, especially anything that is empirical about the paper, full credit goes to Paul.  Anything that seems to be shooting from the hip, that is mine.  So, with that said, we will go through and give you a little bit of feel for what is in the paper.</P> <P>Okay, so our challenge was to look in No Child Left Behind Act and its relationship to research and we were looking at two sides of the NCLB coin, so to speak.  On the one hand, we want to look at this idea in No Child Left Behind that all of the programs funded by it should be based on scientifically-based research and try to understand where that coming from and what the folks on Capitol Hill were trying to do with that policy.  On the other hand then, we are also looking at whether No Child Left Behind itself can be said to be scientifically-based, what rule did research play, in informing the NCLB debate.  Now, that is a huge question.  That can probably take a whole book to answer.  So, for the second question, we did a case study looking at the highly qualified teachers requirement.  There is certainly plaintive overlap with Richard Ingersoll s paper on that.</P> <P>So, this first question is: what were they trying to do with scientifically-based research?  Well, Paul had a great idea which was to look at the law itself and to look at all of the times that Congress put into the law something about scientifically-based research and try to understand if there was some sort of pattern to that or if it seemed rather willy-nilly and sprinkled throughout the law without any patterns.  So, we have some data on that.  And what you find, if you get rid of some of the mentions that are just in the title headers and things like that, if you really look at the substance of the law, there are 216 references to research and 54 percent of the time it specifically mentions scientifically-based research.  </P> <P>So, a couple of things to bring from that.  One is that it seemed like Congress made a decision that some things should be based on scientific research and other things could be simply based on research on a lower level.  You also see that in Title I, which was home to all of the school improvement kinds of initiatives, but, importantly, also home to the reading first program.  It is a place where you see most references to research and the highest percentage that are based on scientific research.  So, there is some method to the madness here.  It does appear that Congress attached research to some places where you could argue where there was some scientifically-based research after all.</P> <P>Now, on the other hand, you do see though that there are plenty of other mentions of scientifically-based research in other parts of the law, including places where I think reasonable people could agree, there is not a whole lot of scientific research.  Look at Title V, for example.  This is where you see charter school programs.  It is also where you see things like the arts programs, parent programs, lots of other things.  When I was at the department, I had the responsibility for innovation and improvement and just a lot of domains where the research basis are pretty thin, and I think you would have a really hard time arguing that there was scientific research in a lot of those areas.  </P> <P>So, you say, okay.  So, why was it, then, that Congress was putting this language in those parts of the law?  Well, we did some interviews with the Hill staffers, and former Hill staffers, and we heard a very interesting story.  Perhaps you might call it a disturbing story.  It went like this.  The Democrats would propose a program.  They would say, We think so and so programs should be in No Child Left Behind.   And the republicans would say, Well, okay.  But we should attach the language requiring that anything funded by this program be based on scientific research.   And the Democrats would say, Well, let's be honest.  There really is not scientific research in this area.   And the republicans would say, Oh, is that right?  Well, then, I do not think we should include this program in No Child Left Behind at all if there is no scientific research.   And the Democrats would say, Fine, include the language. And that is how the sausage gets made.  Pretty interesting.</P> <P>Now, again, when we talk to people and ask them, How can you sleep at night when you make a policy like this?   On both sides, there was some admitting that they put the language in areas of the law where there was no research and they knew that would be a problem for practitioners who are being held accountable for only using programs based on scientific research where there is no research.  But they did say, Look, we are trying to push the envelope, not just on practice, but also on research.  We are trying to create demand for scientific research in some areas that did not have it.  We are also trying to change the conversation as to what they are saying, so that this idea that interventions and education should be based on scientific research took hold.   And there is some evidence that change in the conversation strategy has worth.  </P> <P>You see here another nice chart showing that mentions of this phrase, scientifically-based research, were very few and then all of the sudden, around 2002 when No Child Left Behind comes in, you see this big spike and it has stayed there at least in Education Week and to some degree up and down in some of the major US newspapers since then.  So, there is at least some evidence that this idea is at least out there in the media and being discussed.</P> <P>So the next question was, is No Child Left Behind itself scientifically-based?  One way to look at that is to look at the hearing process.  We were able to go back and look at the legislative history, which the folks in Capitol Hill considered to be from 1995 to 2001.  The legislative history of No Child Left Behind imparted a great job of coding all of the witnesses that came before Congress to talk about this, and quoted them by these different categories And so what do you see?  Well, researchers, being from the research profession, will not appear more than other groups necessarily.  They came in at third place after local level and after group witnesses.  </P> <P>Now, I should point out that groups and individuals do include some think-tanks, Fordham would probably be included in there with advocacy groups - so some groups that you can consider researchers maybe.  But the researchers had a seat at the table and, 17 percent of the time, they were the ones testifying.  And you do notice that the hearings that they testify at tended to be smaller.  So, it does look like Congress likes to hear primarily from local officials local educators tell us how things are going on the local level.  Those tend to be bigger hearings, though, maybe the field hearings or any kinds of hearings.  The researchers seemed to get a little bit more of a hearing, but not much.  But at least they are at the table.</P> <P>But, now, let's look at a particular case study: highly-qualified teachers.  We wanted to see what impact research had in one area, and how that research came about.  And what we argue in our paper - and this is going to feel very self-serving in a moment - is that there were three organizations that we call synthesizers that play the key role in turning research into policy in this debate.  The National Commission on Teaching and America s Future, Linda Darling-Hammond s Group, Education Trust, and yes, The Thomas B. Fordham Foundation.  </P> <P>And what we explain in the case study in the late 90s is that all of these groups came out with reports, manifest, studies, commission reports, on teacher quality.  When you look at these reports, and you look at the kinds of research that they point to, a lot of that research, and certainly the particular researchers that they highlight, tended to have quite an influence on the No Child Left Behind debate around teacher quality.  So, for example, you look at this witness list.  There was a hearing in the house on teacher quality.  A lot of these names will be familiar.  </P> <P>And a lot of these names had connections to at least one of those three groups or at least had been promoted by one of those three groups.  So for example, Gene Hickok, at the time secretary of education of Pennsylvania, had signed on to the Fordham manifesto on teacher quality around that time.  Same with E. D. Hirsch.  You see, Richard Ingersoll said that he had been promoted certainly by Education Trust by the other organizations as well.  Kati Haycock is on there herself and Barnett Barry from NCTAF is on there himself.  And we followed through and tried to look at it again: What are these organizations pushing for and can we see relationship of what ends up getting into law? In fact, it does appear that when there was areas of agreement between these organizations, that seemed to influence the debate around teacher quality.  </P> <P>There was a lot of disagreement.  The Fordham manifesto, I will admit, was very much meant to be contrary to the NCTAF report.  It was meant to be an alternative way of seeing this problem and yet there were some areas of agreement that teacher quality matters a lot, that teachers are inequitably distributed, and that subject matter knowledge is very important.  A lot of disagreements about teacher certification, lots of disagreements about who should make decisions about who is qualified or not qualified.  </P> <P>But on these very high-level 30,000 foot view questions, there was some agreement and these groups, representing constituencies from the left to the right, had said this stuff is important.  And lo and behold, this is where we see some of the key parts of the highly-qualified teacher provision for better or for worse, certainly, the requirement that teachers demonstrate their subject matter knowledge that had some cover from both sides of this debate.</P> <P>What can we take from this particular case study and from these lessons around research base in general?  First of all, when we look at No Child Left Behind, we can say that we think the research did add some substantive value to policymaking.  It was certainly a part of the dynamic.  The folks on Capitol Hill were concerned about scientifically-based research and they did pay attention to some of these studies in the field of teacher quality that came through.  However, and this is obvious, members of Congress and their staff tend to gravitate toward studies that backed up their own ideological views.  So, we heard staffers for Senator Kennedy talk about looking to NCTAF and Education Trust.  We heard Republicans talking about the importance of the Fordham studies in the administration once we are in the implementation.  </P> <P>Again, the Fordham work and the work that researchers at Fordham had promoted, had a lot of play.  So, we certainly see policymakers seeking out their ideological soul mates and taking advantage of the researchers highlighted by those folks.  And the synthesizers, again, to be honest here, in many ways were promoting their own agenda based on their own missions.  This was not a process where all these groups were looking at the evidence and trying to make decisions based solely on what is the evidence says. Ideology, agendas and missions, play a role, no doubt.  But when the synthesizers found the consensus, they help set the stage for bipartisan action.  </P> <P>So, what can we take from this?  Again, I guess on policy I like recommendations.  Researchers, how do you get your work noticed?  Well, one strategy is to seek out the synthesizers like Education Trust, like Education Sector, and others, whose mission and agenda and ideology might find your findings.  That is one strategy.  Another way is, of course, to try to circumvent these groups altogether by reaching out to policymakers directly through op-eds, policy briefs and the like.  </P> <P>And finally, policymakers need to be aware, of course, that people are coming in with these various agendas.  They need to ask these synthesizers and also ask the researchers, come clean about the research, and just ask a simple question:  What would your critics say are the weakness of this research?  Can you bring that to us and make sure we are getting the full story?  Thank you.</P> <P>Frederick M. Hess:  Okay, thank you.  Roberto?</P> <P>Roberto Rodriguez:  All right.  Well, I'm going to attempt to place Mike and Andy s comments into a little bit of a legislative context relative to NCLB up in the sausage-making world, so to speak.  We would like to consider a little bit more of an art form rather than sausage-making.  But call it what you will, it is not perfect and it is very dynamic and fluid.  But my first point really is a very basic one, which is that research is good.  And we as policymakers should do all we can to really listen to research and really struggle and wrestle with these questions of how to better link research to policy and practice in legislation.  </P> <P>The definition of scientifically-based research or the references and statutes to scientifically-based research, while receiving a lot of attention with the passage of No Child Left Behind, really were not wholly new to federal education policy.  But I would certainly say that the shift to really focus on scientifically-based research and to focus on that in the context of the respective federal policy programs is a very positive thing.  Congress has a real desire and interest in investing in what is proven effective practice.  And so, the scientifically-based research definition or the research-based definition or the link to evidence or any of those iterations is a productive one because it really reminds Congress that we need to make sure that, as we invest even in very broad programs like Title I, we are keeping our eyes on what is proven effective, what is good practice.  </P> <P>I think it also is a valuable tool and that it serves to shift the scope and the focus at the program level down at the ground, on really asking the questions: What type of program am I putting in place? What is the basic tenet and the kind of underlying framework of research behind this program?  And one theory of action is that, as those questions are asked more and more in the field, programs will hopefully begin little by little improving upon themselves to a point where they will be more effective and will give Congress a better footing in terms of advocating for strengthening and supporting those programs in the future.</P> <P>My second point is really looking at No Child Left Behind today, where we are going with it with respect to the current architecture and framework, and the challenges as we move it forward.  I would argue that No Child Left Behind does a very good job of assessment, accountability, and standards.  Obviously, those are variable from state-to-state, and No Child Left Behind does not do a perfect job with any of those things.  But the basic architecture there is a good one.  </P> <P>I think now, we are in a place where we have a number of states that have identified over 9,000 schools, Title I schools, that are in need of improvement or corrective action or restructuring and we are kind of coming back around to this question: What do we do to sustain and build capacity for change in our lowest performing schools?  </P> <P>I would argue that this is one of the central challenges and points that we are really going to need to wrestle with as we approach reauthorizing No Child Left Behind and it is a point that research lends itself to very well.  It is a point around how you sustain changes in capacity and teaching and in learning.  It is a point around looking at how data is used in schools and what are the systems in place to support the use of data whether that is looking at formative assessment, whether that is looking at building data systems in the state or at the district levels, whether that is looking at practice in terms how data is examined and reviewed and also how we grapple with questions of professional development and questions of leadership.  </P> <P>All of those arenas are critically important to kind of achieving our goal and our basic premise behind No Child Left Behind of closing the achievement gap and improving public education and those are all areas where research can serve really valuable function and one where we are really going to need to pay a lot of attention to in this upcoming review of NCLB.</P> <P>I would like to just make a few cautions.  Well, I think the shift to looking at research in the context of policy and in the context of NCLB as a really positive one.  There are a few things that I think in Congress we need to keep our eye on and really pay attention to.  The first is really this question of research quality and methodology.  And Mike and Paul s paper does a great job of breaking down these distinctions and differentiations between the statute references to scientifically-based versus regular research-based or even other terms like proven and effective in certain places.  </P> <P>We always have to ask ourselves in Congress, what is the existing base of evidence out there to support our policy of including scientifically-based or including research-based language?  We need to ask ourselves, does that exist now or is that something that we seek to build?  I think there is a compelling interest in certain places.  If it does not exist, a certain type of research, maybe that is an experimental design research or control group treatment group research.  Sometimes policy can serve to drive that forward and I think certainly that has happened in the past few years.  </P> <P>But I also think that where evidence is scarce, we really need to exercise caution in terms of holding out a research quality standard and a methodological standard that we might not be able to attain.  So, that is one kind of tricky question to think about from our perspective.  </P> <P>The second is really to look at this question of research as a proxy for interest and for ideological interest.  And here is an area where you have a number of different instances, both at the federal level as well as the state and even local level, and I think this is just kind of a natural function in some respects where folks are going to gravitate based on their political interest in certain places to certain types of research.  That is something to be expected.  But when that crosses the line in terms of some type of political or financial interest very explicitly attached to research, we run the risk of undermining our more basic goal of making sure that research drives good decision making and good practice.  </P> <P>So, in this upcoming reauthorization I think will be really making a priority of ensuring that we can, to the extent possible in federal statute, really safeguard against the attaching of certain stakes or certain interests to research.</P> <P>And then, I think the final piece is really a piece that returns to the question at hand here that Andy began with -- of school choice and of some of these other questions about what the balance is between making decisions and policy based on research or based on compelling public interest.  I think we have to really do our best to balance both as we move forward.  The research based on school choice is a perfect example where there can be a huge debate over the correlation between student outcomes and achievement and whether that actually says that a particular school of choice is better than a school that it is not of choice.  </P> <P>I think we need to kind of step back from some of that research at some point and say, okay, what is the compelling public interest in terms of pursuing a policy and not necessarily let research get in our way in certain instances as we move forward.</P> <P>Frederick M. Hess:  Okay.  Thanks, Roberto.</P> <P>David Driscoll:  Thank you, Rick.  Most of you are probably a little uncomfortable being in a room that has such nooks and crannies. If you are behind the pole, if you are way off in the corner, you cannot even see.  I'm very comfortable as this is shaped just like Fenway Park.</P> <P>As Rick mentioned, I'm a practitioner for 43 years.  So, I look at this issue and it is interesting.  I came early and sat in the panel before us and I m completely confused.  But I wrote the following sentence before I even read Andy s paper or Mike s or some of the others just on my instinct when Rick said he was going to talk about how research affects policy and I wrote the following: The exact science of ideological politics always trumps the inexact science of research.   And it is the truth.  </P> <P>I happen to have gone through a lot of wars and Roberto is going through the wars trying to reauthorize them.  Well, I have been through a lot of wars over the years.  And it is through the wars that you find out that it really is all about ideology and politics and not much about research.  When we instituted high stakes testing, using the same statistics, by the way,  Walter Haney at Boston College and Linda Darling Hammond can come to completely different conclusions because the data is wrong but it did not stopped them from drawing those conclusions.</P> <P>So you read that, in Massachusetts, the dropout rate went up when we instituted high stakes testing. The truth of the matter is it did not.  It went up recently, of course, and immediately they said it was because of MCAS.  Well, 40 percent of the kids that dropped out had passed MCAS.  And so, it has been my experience that you have to try and filter what comes at you and I think both Andy and Mike make this very good point.  Andy said, evidence changes few minds.   I mean that is really kind of sad when you think about it.  </P> <P>So, people keep doing what they are doing whether the evidence shows otherwise. And we ve had the Reading Wars and I did something a little different than other commissioners.  I attended the Reading First three-day workshop in Washington.  What a thriller.  I got a lot of sleep, caught up on my rest.  But I sat through it because I was very interested because it was supposed to be scientifically-based and it was a lot.  For example, we are very proud of our reading excellence program in Massachusetts.  We have had very good results.  But you really think about it, here was a program that relied on some of the testing every spring.  And so in the spring of 2001, we would show that we had significant improvement over some of the testing in the spring of 2000.  </P> <P>What made perfect sense to me is that in between those should be some formative testing, as they call it a dip sticking, of course, right away dibbles became the biggest issue politically in Massachusetts, but it made sense that you would have teachers do a series of form of testing to find out where individual kids were.</P> <P>Charter schools, a great war in our state, in every state I'm sure.  You either love them or you hate them in my state.  And so, the advocates for charter schools consistently come out with reports which show the charter schools are working by-in-large.  The opponents of charter schools come out and report that they are not working by-in-large.  And they re using my statistics.  Both of them are using the same set of statistics.  It is unbelievable.  And if you think that is an exaggeration, a week ago, two weeks ago, a superintendent who is going to Kentucky, thank God, who hates charter schools - and by the way, we only had about five kids that are part of this regional charter school - he saw it, too, that the Massachusetts Association of School Superintendents put out a report showing that charter schools really are cherry-picked.  They do not have the same percentage black and Hispanic kids, poor kids, as do their resident districts. And on average that is true.  </P> <P>The charter school association put out a report the very next week, knowing that report was coming out, indicating that the charter schools are performing their receiving districts.  The charter school this time got the publicity and that school superintendent did not.  So, it really rests to me as a local practitioner - and by the way a mathematics major - to try and pay attention to research and figure out what does makes sense and, by the way, what you can sell.  I happen to be on the commission for Tough Choices or Tough Times that was put out by NCEE.  </P> <P>One of the recommendations in that report is for the states to go to a defined contribution retirement system because the current system in many states is a geometric progression in terms of its costs. But I can tell you when Massachusetts, it is a non-stopper because of the unions, et cetera.  And so, it really is left for the practitioners to kind of figure out where we are going to put your capital.  And it absolutely makes a difference for kids and therefore, in my judgment, you cannot back out.  That includes the content knowledge of teachers.  That was actually a little bit of a shock to me because as a math major at Boston College, the education majors took almost the same courses we did up through the senior year and then they did student teaching.  But they took differential equations and linear alegbra, calculus, et cetera.  And I slept through decade of the fact that people really never even took a math course if they were an elementary major, for example, even in Massachusetts.  They took math to teach the learning disabled or something and nothing to do with mathematics.  </P> <P>So, I think there are those issues - and when you realize that today you have to do something about them and I think the content knowledge is something that I mentioned the formative testing.  There is certainly enough research to show some of the characteristics of effective schools.  I joked with the Milken Family Foundation, Lowell and Michael Milken.  Michael is focused, really, on healthcare although he pays attention in education.  Lowell is focused on education and they put a barrel of money into both issues.  I tell Lowell, Michael cares very much about prostate cancer, which he suffered from and just put a lot of money into and yet instances of prostate cancer have diminished remarkably.  In other words, the impact they have had has been remarkable.  With all the money Lowell has put into it, I do not think he has made an impact on education yet and he will admit that even their charter program recently showed results from a study that it does make a difference.  </P> <P>But then, again, it is a little bit of the Westinghouse effect because you walk into a school and you say, here is some money, how would you like to try this program.  They all work when you do that.</P> <P>Finally, I would just say, how am I doing?  I haven t even had a sign.  I'm ahead of you guys.  He will get me the look.  </P> <P>Finally, I just think it is up to all of us to stand strong on what we should be believing in.  And when we talk about letting the media be the translators, and I agree with that,� and when we talk about the play that Michael talked about in Congress between the democrats and the republicans on the scientifically-based research, we are all sleeping through that.  We all have an obligation to stand forward and stand up for what we know is right.  Now, I wish my people would not do that all the time.  E. D. Hirsch just wrote to us last week and said, I'm going to come out with a paper that shows the Massachusetts did a remarkable job and they proved it.  You have a statistically significant difference.   And my people rolled back and said, Well, remember the trend analysis.  The accommodations were not exactly the same in the two years.   So, if you really correct for that, I'm writing to my people and say, Let them go, let them go.   </P> <P>But the truth of the matter is that is what we are to do.  We are to stand tall and strong for what does work what we know from the research.  I'll just end with this.  Probably some of you got this on your BlackBerry.  I just got it five minutes ago.  There is a new study out in Education Next by someone from the University of Michigan.  Jacob and someone from Brigham Young funded this study.  I have no idea what to do with this.  But here it is.  Parents of students in high poverty schools value teachers who raise achievement.  At more affluent schools, parents prefer teachers who keep students satisfied.   So, there you go.</P> <P>Frederick M. Hess:  Dave, we got copies of the new Education Next.  It is out in the lobby.  Help yourself to a copy.  It is Brian Jacob and Lars Lefgren.  And Marty here is happy to explain the study in length.  Anybody who has got questions about the article.  Andy, Mike, comments, doubts before we open this up?</P> <P>Michael J. Petrilli:  I do want to make the comment that I do not think is quite as bleak, perhaps, as it has been sounding at the last couple of panels.  I want to point out that when we look at No Child Left Behind, again, it does look like researchers.  I had a place at the table and some of that research informed what went into No Child Left Behind.  So, that is some good news.  </P> <P>But the other thing that I keep hearing and I think is very right is that research, more than anything else, motivates people to do something. In fact, David used that phrase just a moment ago that we looked at the research on the importance of subject matter knowledge of teachers and we have to do something.</P> <P>At this point, where research ends and where ideology and politics tend to take over, is where we get into the details.  And maybe that is how it is always going to be.  I mean, if you look at the highly qualified teacher s provision, we can all look at the fact that subject matter knowledge is important and that poor kids tend to have teachers who do not have as much of it.  And we can all probably come up with five different solutions to that problem, none of which perhaps have been tested specifically yet by good research, but are based on our instincts about how human beings operate and how organizations operate and the appropriate role of the federal government and if that is ideology, if that is instinct, and again I do not know that we are ever get away from that.  </P> <P>But I think the good news is that if you come out with some research that demonstrates where there is a problem, and especially if it lumps into our feelings of social justice that there is a problem in that poor kids or minority kids are getting a short end of the stick and that can motivate policymakers on both sides of the isle to do something, is where we get into the details that we tend to run into trouble.</P> <P>Andrew Rotherman:  I guess, I think it is bleaker.  And I say that is sort of the themes I have heard here.  I think in education for a variety of different reasons there is sort of a mode of appeals that carry the day over empirical appeals.  It is just it is the way it is.  And there are probably lots of reasons, some of which we can talk about at a conference, like fundamental disagreements around research methodology, what constitutes high-quality research in the first place, I mean, you see that around a lot of these issues and some things that are endemic to our field and probably things that I think are also just sort of characteristics of political debates.  </P> <P>So, I think what I tried to lay out, and I'll be as articulate about it is: you have two questions.  I think one of them is fundamentally ideological question and that, if we treated it that way, we would actually have a richer debate.  It is a fascinating question.  The other is an empirical question.  The effects question is a very important question.  And after we have that ideological debate, then you have decided what you are going to do and you get into all kinds of really important secondary questions around effects, where research really can tell you things.  </P> <P>I mean, there are all sort of things you have already learned from school choice, the demonstration projects, and so forth thus far, that we can then apply to making a policy and so forth.  And you see that on a range of issues whether it is the teaching issue or the reading issue and sort of -- it will be great if science carried the day, but it is not going to.  But if we can at least be better at disentangling those things, I think you end up with better policy and, frankly, a more constructive and interesting conversation.</P> <P>Frederick M. Hess:  I mean, it seems to me, Andy, I'll ask you to elaborate on this a bit, we have only talked about the medical research community, for instance. I mean, it does not strike me to very frequently that NIH is deciding whether to allocate resources to one area or another based on some research out there.  I mean, right off with these concerns that, in fact, areas of inquiry are directed by a whole bunch of fundamentally irrational things.  A senator s child has an illness, or it is an influential advocacy group.  What we are able to do, of course, is figure out how we address a problem once we have set our minds on it.  We are then able to bring research to bear when we try to invest resources in ways which generate more useful and reliable answers.  I mean, it sounds to me that part of what you are suggesting is that we have got a fundamental confusion when we talk about whatever research can and cannot accomplish in the policy arena.</P> <P>Andrew Rotherham:  Yes, definitely.  I mean there is irrationality everywhere in this.  There are idiosyncratic examples.  That exists across the board and that is a characteristic of politics.  I do think you see in some fields, whether it is because you have sort of uniform appropriations or a sort of more centralized structure, you have a more coherent research and development agenda than you see in education.  It really is all over the place, the exchange that you and Michael had this morning, is illustrative of that -- the lack of consensus.  So, yes, I think that is exactly what I'm suggesting.</P> <P>Frederick M. Hess:  Okay.  You go ahead and open it up.  Steve?</P> <P>Steve Fleischman:  I am Steve Fleischman from AFT, I mean AIR.</P> <P>Steve Fleischman:  I'm surrounded by my former colleagues.</P> <P>Andrew Rotherham:  It is a d�j� vu thing.</P> <P>Steve Fleischman:  But I have been wondering about this question since I was AFT, which is -- essentially, what I have heard from everybody is politics trumps evidence every time if the evidence does not conform to the political stance.  So, I'm wondering and you have all served publicly elected officials -- can you imagine any kind of pledge that a politician might take that might be public that would in any way kind of mitigate this instinct to sidestep the evidence when it was clear that the evidence was pointing in a direction that the politician did not want to go?</P> <P>Michael J. Petrilli:  I think that will be a really hard sell, yes.  But I think, again, I want to bring us back though to remember that we are talking about policy and not practice.  And once upon a time this promising young scholar named Rick Hess wrote a great piece in the Educational Gadfly that argued that in the medical world, we do not expect scientific research to answer the question of should we have HMOs or not. Some of these questions about how we organize education delivery are going to be harder to answer scientifically than whether a particular reading program raises achievement for a particular kid.  </P> <P>And so, I just think that when we talk about policy, we need to admit that we are never going to get ideology out of it altogether, nor should we.  I mean, Andy did a great job on the choice debate separating the two questions.  I would say on the teacher-quality debate, so much of it comes down to who you trust to decide whether a teacher is qualified enough to instruct the group of students?  </P> <P>And one side of the debate, people say, Well, I'm going to trust the principal because I think that person has the right incentives, yada, yada, yada.   Somebody else might say, No, I would trust somebody at a central level to make that decision instead of the principal.   I mean, how do you disentangle that from evidence?  I do not think it is fair to ask policymakers to make decisions based entirely on evidence because I do not think that is ever going to be completely possible.</P> <P>Frederick M. Hess:  Although let's push a bit more on Steve s question.  So, for instance, when we think about the Federal Reserve, it is certainly the case that, especially when looking at a national election, demonstrates we always would rather have interest rates lowered.  And we seemed to have created machinery where we insulate the federal reserves.  So if their professional judgment is that it s in the national interest not to cut interest rates, they seem to be protected to some degree from short-order political interest.  Is there anything that we could import from those insights in the education, which might help to address some of these tensions we are talking about?  David?</P> <P>David Driscoll:  I think there are a lot of examples where people have stood politically strong, but there are more practical issues rather than policy issues and again ideology seems to trump.   I mean, the people have stood strong against bilingual education, or whatever the issue may be.  I think this idea that people can come together, maybe even Mike and Andy sitting next to each other, but putting some of the people that have been at war together.  Get the outliers out of here, get the people out that are really lying out of the ballpark, and start to get people and start to get some consensus.  </P> <P>I think that could help because no conclusion is a better answer than to give a false answer, which I believe some of the people do, as it was pointed out.  You can see an article written and then they go to the other side and you see you get this -- I think it is up to the research community to come together to get out the phones and try to get some consensus, for example, in mathematics.  I do not know if they have solved the issue about how to teach mathematics.  But at least we ought to know that it is about both the content, the basics if you will, and also the motivation to teach.  I mean, that is a common sense.</P> <P>Andrew Rotherham:  Yes, and I think you put your finger on something and, yes, the answer is you can do things to insulate things from politics and there are things you can do in government.  There are things you can do.  They are sort of extra-governmental.  The research community can go a long way toward it this is what I was talking about earlier with the media.  </P> <P>There are steps you can take to definitely insulate things from politics and there are lessons we can draw from other fields and so forth.  It does not have to be like it is now with all those things where they have in common.  All those things you can do to change the incentives or to be more blunt in the data and the language in politics.  Things you can do to create political cover for people to make decisions, but, again, it is an ideological space because these are very fundamentally unsettled issues.</P> <P>Michael J. Petrilli:  And I just wanted to say that we heard earlier this morning from Andy that the current IES structure, the What Works Clearinghouse, that all of that, I think you could argue, has been insulated from politics.  I mean, it was remarkable that the What Works Clearinghouse came out and declared reading recovery to be effective at least in some ways at the same time at the administration very clearly has not determined that program to be effective.  </P> <P>And so, I think there are some ways to structure that.  And if we were honest in our Reading First debate, we would say, look, instead of crucifying public officials who give their heart and soul to implement a program in line with what congress has asked them to do because they have tried to follow through on what the evidence is, we would say, look, if this is a contentious issue, let's have a federal body, make a decision, make a list of the effective reading programs and just help people use that and have some kind of FDA for research, an FDA for education, and be honest and be open about it.  Instead of this situation where we -- we do not have a list, but there is a list -- and we are going to crucify the people who try to implement the program in a way that they see fit.</P> <P>Susan Ott:  I'm Susan Ott with the Milton Friedman Foundation and my question is for Mr. Rodriguez.  Can you take that tradeoff you discussed about compelling interest versus research-based and place it in the context of limited resources?  And are you concerned with whether research-based policy might cost more or create unfunded mandates like has been claimed about highly qualified teachers, I mean, relying on the research to say, we need to put the money towards this because the research supports it?</P> <P>Roberto Rodriguez:  I mean, clearly, I think research and the fact that you bring research into certain decisions or certain particular policies carries with it financial implications.  I mean, I do not think I would deny that.  But I think my broader point was just to say, there are certainly, I think, compelling public interests, where the research will either be thin or inadequate in some respects or another respects there will not be a research consensus and that still does not mean we should not follow through with a particular policy.  </P> <P>I mean, I think No Child Left Behind honestly is a perfect example and we can talk a little bit about whether there was -- we listened to certain scientist and researchers in the development of that policy.  I mean, I think Congress did.  But I think the broad goals here of closing the achievement gap, of raising standards, of improving public education, I mean, there are huge bases of research that we can point to.  Those are compelling public interests.  I think we have to have a strong bipartisan congress say that this is the way we need to move forward.</P> <P>Frederick M. Hess:  Roberto, Mike and Paul in their paper, they make the assertion that, look, there is so much research floating around out there.  Folks on the hill have so much on their plates.  There is no way they are actually going to go out and identify research.  So, these synthesizers in DC, these advocacy groups similar to Education Trust, play an important role in terms of identifying who is going to show up on the radar screen and who is going to actually be heard in a testimony and who is going to be invited to meetings on the Hill.</P> <P>In your experience, I mean, is that the way it works.  Would you suggest that there is no re-simplification, what is your take on?</P> <P>Roberto Rodriguez:  Well, I mean, I think it is -- I do not think that is the full picture, but I do think it is accurate.  The reality and the demands on being on the Hill and in crafting policy, and I think both of the papers are really acknowledged this, really show that the capacity to distill primary research is limited on the Hill.  </P> <P>So, we do rely a lot on organizations that conduct applied research or that bring forth some findings and to dig a little bit deeper.  But I think it is really incumbent on every office and particular members to really question where does that research come from, what are the foundations of that research, and is there a particular agenda attached?  And I think most offices do, at least within a committee, do a very good job of that.  I mean, there is not a sense of na�vete in that you are distilling research from an organization and that is pure.  I mean, I think you see a lot of different sides come at this.</P> <P>Andrew Rotherman:  I'll say something because Roberto cannot praise himself.  I think one of the things that I really bristle at about this debate is when you hear that the people on the hill do not know what they are doing.  They are fools and so forth.  I mean, when you get to the committee staff, particularly to the senior committee staff, nothing can be further from the truth.  These are really sharp people who are very smart and, over time, develop an incredible expertise around these issues and, look at people like, for example, a guy like Charlie Barone on the House Committee with some of the assessment stuff.  He has a PhD psychologist.  He knows this stuff and he has got this clown saying how he doesn t know what he is talking about.  He has forgotten more about it than a lot of folks know and that is the upside.  And I think even at the member level, the folks who have been around for some of these committees are really deep on these issues in a way that I do not think people appreciate.  </P> <P>The downside of that, though, is like the rest of us. Then it starts to create blind spots you get set into and you tend to reject things that do not comport with your frame of reference and so forth and so like the rest of us they have those blind spots and so it is a tradeoff.  But I think there is a general underestimating, in the scholarly community and public community in general, of the level of expertise that you see in a lot of these committees.</P> <P>Frederick M. Hess:  On that question though -- So, if we presume the expertise, but people on the hill, staffers as well members, obviously get invested and statutes and ideas, and we have not talked about much, but researchers, even outside of financial concerns, financial misbehavior, researchers get invested in lines of research and in arguments.  This becomes their identity as scholars.  They spent a whole career.  It is very difficult, once you spent 20 years researching something, to change horses and say, the last 20 years, that was wrong.</P> <P>Michael J. Petrilli:  It s easier than you think, Rick, actually.  </P> <P>Frederick M. Hess:  If this is a kind of flexibility you get by not having a PhD.</P> <P>Andrew Rotherham:  Well, I think it depends a little bit by issue.  I mean, there are some folks on other issues.  I think Dan Goldhaber is a great example of somebody who puts out really solid work that people question very often.  He has developed a reputation for that.  I think it is different by issue depending on the politics.  The issue I looked at here of school choice.  You do not see a lot of that because the politics or organization or the funding sources and so forth people are on different sides and you are not going to see any of the big Washington, ed establishment associations and their research shops decide, you know what, these studies do show a modest effect for vouchers and maybe that is a cost-benefit question and maybe that is a good policy.  We are all going to grow old before that happens.  And so the incentives are all against that on certain issues.</P> <P>Bill Hall:  Bill Hall, University of Maryland.  This is a pro-research question and comment.  I think the distinction that people want to draw between ideology and research is a little too neat.  In lots of examples, we see ideology is based on assumptions about outcomes.  Let's take choice.  We have a theory about this.  Well, if that theory gets undermined by research, we see not the people at either end that somebody pointed out, but we see a gradual movement in one direction to another.  </P> <P>And we can use lots of examples.  I think bilingual is a good example of policy that has been more heavily ideological and it is going to be in the future because we know a lot more now about it.  That undermines the argument that is going to be this or that. And we talked earlier about the Reading Wars.  Well, that whole argument is, at least in the informed discussion, is changed because of research.  I do not think there are too many people arguing that it has got to be one way or the other or its not going to work.  </P> <P>So, we have to take kind of longer view of the impact or research on anything where research essentially eats at ideology, ideology certainly informs research and research gets done, as you point out.  So, this is Carol Weiss argument from a long time ago about how research influences policies.  Not in the short run.  But in the longer point of view.  And that means having a more consistent approach to research and a more programmatic one than we tend to have.</P> <P>Andrew Rotherham:  I think that is right, but again I think it depends by issue.  I mean, so, for example, reading. There is no reason why people who think that phonetic-based awareness really matters are somehow conservative, that has always just baffled me, and why whole language is a liberal position.  Things get ascribed to this.  That is one issue and I think there is more some science and some other issues like on choice.  I take the point in general, but I think the issues are more fundamentally ideological than that and there are just big picture distinctions in terms of people s views in the world and those are great.  Those are legitimate.  We should celebrate them and say you are not going to sort of have overtime.  The political constraints that the people operate with may change and that may change their behavior, but you do not see the trajectory that you are talking about on every issue.</P> <P>Frederick M. Hess:  There is an interesting question here, too, which I think is the long viewpoint that Bill alludes to, when we think particularly about foundations in this environment wanting to fund research which is going to have an impact.  It is going to have an impact on achievement in a two-year cycle.  Anyway, frequently, the impact of resources and research is not necessarily in the ability on improved results, six months or 12 months, or 24 months out.  But in the ability to change the way people think about problems and understanding institutions over a 10 or 20 window.  How do we kind of retain the focus we want to have on the immediate goals or NCLB's language to 2014 without short changing our ability to think broadly about what Jeff referred to this morning as investment of basic research and basic research and basic knowledge construction?</P> <P>David Driscoll:  I'm not sure yet.  I'm not sure you get there.  I mean, I really do not.  I'm very much worried about the long term.  I mean, it is all short term.  It is all We ve got to fix it, we can find this, we can find that.   I mean, Gates says, Oh, its small schools.   So we put a barrel of money into the small schools and then they find out it is more complicated than small schools.  So, and of course, Americans do not take the long term.  Now, we are still debating on what the mathematics curriculum should be.  Singapore figured that a long time ago.  And we are still fooling around 50 ways to get to addition.  </P> <P>So, I very much worry about this long-term approach and I do think there are examples where some arguments get settled.  So, therefore the consensus on research kind of moves forward and that is why I urge, if nothing else, I would like to have the researchers get rid of the phonies.  That will be a big help because I still see people out there that are making statements that are absolutely ridiculous and not only do they get printed but they get sought out for their opinion when someone else comes up with something that might even add value.  </P> <P>So, I think this issue of trying to get a consensus, and you talked about the Federal Reserve and I do agree with the IES, I think we have got to come together and say let s knock this stuff up recognize its complicated and try to build slowly, but surely, a field of research that is honest and it will take the long run.</P> <P>Andrew Rotherham:  But I think, to your point, the researchers, they do not shout their views.  Reporters call them and so forth.  They look for the cues and the difference between how education research is reported and now, for example, medical research, is its really striking.  Medical research stories tend to be authoritative and tell you how it fits on literature.  Education stories, you get a finding and then people s opinions about the finding in the back and forth, so to get these balanced stories and an instructive example is Russ Whitehurst.  I write about this in the paper.  He offered to help reporters with a quick turnaround service.  Confidentially, they could send in any study that had come out.  They would give a quick confidential review on it in a 48 hour turnaround.  This is a service that the government is going to provide for free and he is gone exactly no takers of the last three years on this.  And so, there has got to be a greater awareness to help for what they are talking about.  These people are doing with the incentives are there for them to do, which is to get in the papers for their funders.  The level of debate house has to increase all around.</P> <P>Frederick M. Hess:  I mean, Roberto and Mike, I mean particularly in the notion where one of the bright lines for reauthorization is 2014.  In this environment, it becomes very difficult for folks to say, well, this is not going to show any payoff until 2017, but it might be useful then.  Of course, Big Pharma, the drugs that Pfizer is investing in today is, they do not really expect that the trials they are initiating today are going to actually deliver the product before the 2020s.  It takes some years to get the stuff done, then they ve got years of trials and years of approval, and it is just understood that that is the cost of doing business.  </P> <P>And so, partly, the way you report the story you have an entire intellectual structure.  That is kind of rolled and then place when drugs were gone to approval.  We kind of want that part.  This was kind of the Reading First challenge.  The good stuff of what we think comes from the medical model, but we do not want all the painful kind of having to wait and putting it on the shelf and waiting for Christmas.</P> <P>Roberto Rodriguez:  And I think we have to set for the fact that it is not perfect.  I think we have to settle for the fact in certain places, and 2014 is a perfect example, we are not going to have that empirical evidence. The 2014 goal is what it is.  It is a goal.  It is a goal that Congress puts forward to try to move the ball forward in terms of efficiency across all the subgroups.  On certain other questions, maybe we are going to have the research and the empirical data out there to be able to say, look, this is what is going to back the decision that we make in terms of the policy objective that we choose and we should expect to see results and we should hold ourselves accountable to that research.  </P> <P>So, I mean, there is such a gray area there and it depends on the particular policy that you are looking at within NCLB, which goes back to why in certain places we have scientifically-based that is referenced in other places we have researched-based.  In other places we have to be careful and say, look, do we really want to turn the research around this particular policy objective or is this a policy objective that is consistent with a general thrust of where we want to go is a bipartisan Congress and is an administration?</P> <P>Frederick M. Hess:  Mike?</P> <P>Michael J. Petrilli:  Well, I just say that I am more optimistic when it comes to researching specific interventions than reading programs and math programs and the kinds of things that schools and districts need information on.  I think that we have started to care about some space for those long-term evaluations at IES.  This will be more funding.  It is still early.  We will see.  But I feel more optimistic.  The policy debate is going to always be messy and it is going to be -- that is democracy.  Some states are going to try things that people will be interpreted in different ways and things will happen in a messy way and I do not think that will ever stop.  I do not think that is necessarily though the worst thing in the world.  We just need to make sure that research is informing that process and is at the table, but it is never going to be the only thing that makes the decisions nor should it be.</P> <P>Frederick M. Hess:  Okay.  We will make that the end of the panel.  I want to thank you all.  </P> <P> </P> <P>[Start of Panel 4:  How Research Is Used by the Public, the Courts, and Educational Leaders]</P> <P> </P> <P>Frederick M. Hess:  Okay.  We are now gearing up to kick up the fourth panel.  This panel is going to focus on the way that research is utilized by the courts and by educational leaders and by the public.  Here, there is a slight wrinkle, which you get heads up for because you are in the audience.  For the book to come out of this conference today, you are hearing 10 papers presented.  There will be an eleventh, authored by William Howell.  He is using newly collected public opinion data which is not yet available, so we do not yet have William s paper available for you.  We will be disseminating that paper.  It will be available we expect late August, give or take.  And so, it will be on the website with the others at that point.  I think it promises to be really interesting and will certainly be included in the book.  But unfortunately, we are not able to talk about those findings today, or I guess those future findings today.  I get confused by my tenses on occasion.  </P> <P>With that in the last two sessions, we really try to speak about how research plays out in specific policy debates and specific policy contexts.  What we want to do now is broaden it out a bit to talk about how other players and institutions utilize education research and the way in which they influence the production of education research.  The two authors presenting on this panel:</P> <P>First, we have Josh Dunn.  Josh is an assistant professor of political science at the University of Colorado, Colorado Springs, where he teaches courses on Constitutional Law.  He is a coauthor of a forthcoming book, Complex Justice, from University of North Carolina Press on the case of Missouri v. Jenkins, and he coauthors a quarterly column on law and education for Education Next with Martha Derthick.  </P> <P>Speaking second is Lance Fusarelli.  Lance, we ve got you on another page here.  Lance is associate professor in the Department of Educational Leadership and Policy Studies at North Carolina State University.  He coauthored Better Policies, Better Schools:  Theories and Applications and recently co-edited The Politics of Leadership:  Superintendents and School Boards in Changing Times.  </P> <P> Our discussants:  Speaking first will be Pat Forgione.  Pat was named superintendent of the Austin Independent School District by the board of trustees in August 1999.  Prior to that, he served as US commissioner of education statistics with the National Center for Education Statistics.  And before that, he has a long and checkered history, which is familiar to many of you.  </P> <P>Speaking second is William Howell.  William is an associate professor at the Harris School of Public Policy at the University of Chicago.  William is a principal coauthor with Paul Peterson of The Education Gap:  Vouchers and Urban Schools and the deputy director of the Program on Education Policy and Governance at Harvard University.  </P> <P>And then, the third discussant is Warren Simmons.  Warren, as many of you know, is executive director of the Annenberg Institute for School Reform at Brown University.  Prior to joining Annenberg in 1998, Warren was executive director of the Philadelphia Education Fund, where he supported district-wide efforts to enact standards-based reforms.  With that, Josh, would you please get us started?</P> <P>Oh, and actually, while we get you set up, I understand that we are out of our little friendly CDs with the papers.  So you cannot grab those.  But again, please feel free to grab the hard copies of the papers in which you are interested.  They are all available right outside.  And again, all of the papers from this conference are available online at <A target=_blank href="http://www.aei.org/" target=_blank>www.aei.org</A>.  And for those of you who are truly seeking and entertaining the opportunity to relive this day, this thing is, like all AEI events, being webcast.  And you can always watch it all in full color on your computer anytime you want.  All right, Josh. </P> <P>Joshua Dunn:  Thanks, Rick.  First, let me recognize my coauthor, Marty West.  He was initially not going to be able to here, but his son decided to arrive early.  So, he is a new father and was able to make it down now.  So Marty -- any difficult questions I ll pass it on to him.  </P> <P>Well, typically with a paper on this topic, you start with a quote from Alexis de Tocqueville about how every political question in the United States ends up becoming a legal question or a legal issue.  I think one of the great strengths of our paper is we actually resisted using that quote.  We did not include that.  But there is, of course, some truth to that.  And the difficulty for us was, with education policy and education politics being such a vast area, it naturally has led to an immense amount of judicial activity in the area of education.  So we tried to get a hand or grasp on how it has been used in different areas.  We decided to focus on a couple of areas where education research has been used most often.  That is desegregation and school finance litigation.  Before we get to desegregation and school finance litigation, I will go through some of the background issues so that we can make sense of how this research is used.</P> <P> The first thing that is immediately apparent when you look at the use of social science, education research, in court is that its use has increased dramatically, starting with Brown versus Board of Education on through today with adequacy litigation, which I ll talk about in a little bit.  It is essential, in fact, to adequacy litigation.  So there has been a marked increase in the use of education research.  It also has had a varying role in how it is used.  It often will depend on what stage of litigation you are in.  </P> <P>So, for instance, if you are determining whether or not there has been a violation of a right, you will have a different use of education research.  If there has been a violation that has been established, what would the remedy look like?  And, of course, at the appellate level, the research that you use will be different in what you use at the trial level.  </P> <P>You also have an inconsistent application of education research in the courts, and in fact, social science research in general.  Courts of last resort, particularly the United States Supreme Court, have no obligation to use social science evidence consistently.  They can use it as they want and dismiss it as they want.  Sometimes, the very same justice does this.  William Brennan in Craig v. Boren said that, Well, you just cannot expect judges to actually understand all of this, that this school stuff that is going on.   This is a case about different drinking ages for men and women in Oklahoma.  But then, you get to McCleskey v. Kemp, and he had no doubts about the social science evidence that was presented to him.  Sometimes you find the very same court, the very same justices, the very same judges, using it differently.   Another background issue was trying to limit admissibility of education research that actually should read unlimited admissibility.  It is almost impossible to limit social science that is present in court, particularly education research.  The Supreme Court has adopted what has been called the Daubert standard, for Daubert trilogy of cases where they tried to limit junk science and how often it gets into court.  So they have established some standards. For instance, can evidence be empirically tested?  Has the method been subjected to peer review?  Does it have a known error rate?  Is the method generally accepted within the relevant scientific community? They do not have to have all of these.  But these are just guidelines that the Supreme Court has established.  </P> <P>If you actually look at those standards, it is hard to imagine how education research could be excluded under them precisely because of the reasons that we have been talking about today.  There is such a massive amount of education research.  There are no clear or established standards for what counts as good education research.  So the practice of judges, when they have entertained education litigation, has just been to admit everything and then decide later what they are going to rely on.  So really, it is almost impossible to limit education research.  </P> <P>And then, one final issue around Federal versus State Courts as Jeff, I believe, mentioned earlier.  There has been this trend of moving education litigation from federal court, particularly with the gradual decline of desegregation and the rise of school finance litigation, and now with the air of adequacy.  So state courts are taking on an increased role, and this has important implications for the use of social science because we argue that state courts are more political.  Their method of selection is more political.  Their method of retention is more political.  And instead of making them less likely to rely on social science, we think this actually makes them more inclined to rely on social science testimony in court.  All right.  </P> <P>So these are some of the background issues that we dealt with.  Very quickly desegregation.  Of course, all of this starts with Brown v. Board of Education, footnote 11.  What is interesting is that the issues surrounding footnote 11 are really still with us today.  The questions that were raised after footnote 11 appeared in Brown v. Board of Education, of course, relying heavily on Kenneth Clark s doll study, is precisely the legitimacy of using social science in determining the meaning of constitutional clauses and also the quality of the research.  </P> <P>Immediately, questions were raised about Kenneth Clark s study and whether or not they actually supported the Supreme Court s opinion.  In fact, segregationists in the South said, Well, if you actually look at the study closely, it actually shows the opposite effect of what he claimed that it did.  And so, we should have segregation.   Then in response, many said, Do you really want clauses of the constitution to be determined, their meaning to be determined, by a changing an unreliable social science?   </P> <P>The next thing we noticed with desegregation is that doctrine had a very important effect on how education research was used, particularly in the aftermath of Milliken v. Bradley.  Prior to Milliken v. Bradley, a lot of research would be presented in court about housing patterns, the effects of different policies that the school board would adopt, and how that would affect attendance boundaries, and then housing, all of that sort of stuff we get into.  With Milliken v. Bradley, though, you had to turn towards compensatory remedies, and this required, absolutely required, bringing in educational experts to design compensatory remedies for desegregation cases.  </P> <P>So prior to that, you have the social science testimony.  But, in many cases, even though it was always there, it was not essential for the court s finding.  In fact, I would argue that most of the time, fairly basic, descriptive statistics were sufficient for the finding of the court.  Most of the time, based on the doctrine that the court had established, they were proper to find school districts in violation of the Supreme Court s precedence.  But after Milliken v. Bradley, they had to design affirmative programs to try and remedy past wrongs, this compensatory programs.  </P> <P>And this eventually led to Missouri v. Jenkins.  This was the high mark, I think, or maybe depending on how you look at it, the low point of the use of educational expertise by the courts.  What happened in Missouri v. Jenkins was that we showed that small-scale compensatory programs, generally magnet school programs, were not sufficient or effective in actually remedying the problems facing students.  So they would have large-scale programs and so they ended up spending $2 billion to try and remedy the problems of the Kansas City School system.  And it did not really work. </P> <P>Then finally, declining importance.  We think in the late stages of this desegregation that educational research is actually not been all that influential.  All right.</P> <P>School finance:  This is where we think most of the education research is being used today.  It started off with equity.  San Antonio v. Rodriguez was important because the Supreme Court decided in that case that the Constitution did not guarantee a right to education.  And so, people trying to pursue remedies through the courts had to go to the states.  So then they look to equity which was equal spending that they said was required by the states that had some problems with it; primarily because equality cuts both ways.  You can either make things equal by bringing everyone up to the same level or by bringing everyone down.  </P> <P>So eventually, they turned to adequacy which claimed that you can actually determine with a sufficient degree of precision what actually needs to be spent in order to provide an adequate education for students.  This is where education research is essential.  You absolutely have to have it so educational experts come in with adequacy cases and try to determine what policies, what expenditures are necessary, in order to reach state constitution s required level of education.  We also noticed that there has been a bit of a recent pullback with adequacy.  Some courts have refused to entertain these cases.</P> <P> So in conclusion, the final points.  We do have concerns about the usefulness of education research primarily because even with rigorous research, it is not clear with these large-scale programs that judges are asked to order, that you can scale them up on any sort of reasonable way.  This actually happened in Missouri v. Jenkins.  There was some evidence that if they adopted these policies that they might work, but the attorney later said, Look, we just had no idea how to scale it up.   Of course, there are remaining questions of legitimacy.  Is it appropriate for the courts to use social science evidence in these sorts of cases?  </P> <P>And then finally, quality as sort of a catch-all.  One of the things we noticed with reading the other papers for the conferences is that many of the problems that have been identified in the other papers, we think, are exacerbated by using education research in the courts.  Particularly, we have noticed, or many people have pointed, out that the relationship between advocacy and research tends to undermine faith or confidence in the research.  And that obviously, is inherently tied when you have extra witnesses going in the court.  It is inherently tied to advocacy.  </P> <P> Frederick M. Hess:  Okay, Josh.  Thank You.  And Lance.</P> <P> Lance Fusarelli:  All right.  In my paper, I looked at the extent to which school leaders, primarily principals and superintendents, use or do not use research in decision making.  And essentially, I focused on four key questions.  Do school leaders use research in decision making?  If so, in what ways do they use this research?  If not, why not?  And then, what bearers exist, and what can be done to improve research utilization in education?  </P> <P> It is commonly accepted that effective organizations use research, that they improve practices systemically overtime, that they, in effect, become learning organizations.  And in my paper, I look a little bit across organizational sectors, in terms of learning in the private sector or the public sector.  It is much less clearer whether schools are learning organizations, oddly enough.  Children learn, albeit unevenly, but it is much less clear whether adults in schools learn.  And so, what I did was review the literature.  I reviewed the professional trade journals going back about six or seven years.  Then I had conversations with 19 different people: superintendents, district level leaders, principals, and program evaluators.  </P> <P>And basically, I found that research utilization among school leaders is very uneven.  They can be informed by it -- whole-school reform models, and teacher turnover, value-added assessment, class size reduction.  But a lot of those tended to be imposed upon schools.  It is not like school leaders get together and say, You know what works.  And what should we do?   Some of this is externally driven and imposed upon them.  There is increasing evidence primarily because of state accountability mandates and No Child Left Behind that school leaders are using more data-based decision making.  And that is different than how traditional eggheads think about research.  It is more action research.  It is much more contextual. But they are using data more and more in terms of decision-making.  </P> <P>Now, then the question would be, why are they not learning more from us, from the experts?  There are a number of significant structural and personal barriers that I uncovered talking to people about this.  The conflicting findings have been elaborated by a number of other panelists.  </P> <P>Christopher Cross, formerly assistant secretary of OERI called this a Cross s corollary.  For every study in education research, there is an equal or greater number of opposing studies.  Have you ever seen the excellent Kevin Bacon movie He Said, She Said?   That is how this looks to a lot of school leaders.  They see studies and then the next week, they see a study that says something else.  And what they want is somebody to say, This works.  This does not.  Use this.   </P> <P>And they do not get a lot of prescriptive help from the academic community from policy.  There is a lot of poor research.  There is also a lot of skepticism and mistrust.  If all the educational researchers in America die tomorrow, there would not be that many educational leaders that would come to our funerals.  They tend to be very skeptical.  They are.  Truth cannot understate this.  They are very skeptical of the purposes of research.  Many of them, not all of them but many of them, view this as advocacy.  They see it as an increasingly politicized environment.  </P> <P>The fact that you have a wider dissemination of research and technology bypassing a peer review contributes to that policy noise because they are looking and they are seeing all of these information sources, many of which can be contradictory.  And they do not have a lot of time to sort through this stuff.  And it is a problem for them.  They can be very jaded with it.  And the media, by playing up these debates, only adds fuel to the fire.  They do tend to rely on their professional expertise a great deal, a preference for context, local context.  If there is a regional diffusion, it is generally talking about what their colleagues and neighboring districts are doing or whether or not that is working.  </P> <P>There was a story in EdWeek a couple of weeks ago on Los Angeles Unified.  The chief instructional officer admitted that she did not look at any of the research about the Waterford Early Reading Program before LA Unified committed $50 million to the program.  They did two internal evaluations of the program and found out that not only was it not increasing student achievements, it was widening the achievement gap.  And when I asked her about it, she said, Well, basically, I listen to my teachers.  And I listened to my context as opposed to abstract statistics and meta-analysis.   And that approach is not atypical in decision making in school systems.  So you have the politics of it that have already been discussed.  I call it personal beliefs.  I m using a more polite term and ideology, but that has been discussed by several other panelists at length.</P> <P>Another issue is lack of time and lack of expertise.  If you are in a constant crisis-management mode, where a good day for you as a school leader is when nothing bad happens, it is kind of hard to have all that extra time in the day to engage in systemic strategic planning and sift through all the research to see what works.  Those types of things, when you are operating under incredible time constraints and incredible pressure can be very difficult.  There are others one of the superintendents in my study, a couple of them, commented that because of this pressure, schools lack a culture of inquiry; they simply do not have time for the type of reflection that is necessary to engage in and implement research-based practices in school, particularly, if they have to sift through a lot of information.  And there is a lack of communication between educational leaders and educational researchers.  Sometimes what we are interested in is not what school leaders need research on.  </P> <P>I had a departmental meeting a week ago, and we were talking about research programs for our doctoral students.  Someone suggested, Well, why do we not ask the school district what they want researched on?   And literally, all hell broke loose because they were like, Well, they cannot take it.  We do not want them dictating our research agenda to us.  And they are students.   And it went on for about an hour-and-a�-half.  I lost interest after about five minutes.  But it was very contentious.  And there are issues about that with respect.  There are issues about legitimacy.  </P> <P>So, I wanted to kind of conclude with some positive things.  In the first draft of this paper, I was completely pessimistic.  I thought the situation was hopeless.  I decided under Rick s guidance to become a little more helpful.  So here it is.</P> <P>Frederick M. Hess:  That is the first time I have been a ray of sunshine.</P> <P>Lance Fusarelli:  Yes, I know.  You are a ray of sunshine.  I think No Child Left Behind is helping in the school accountability system.  I think it is helping schools to become more data-based decision makers.  Schools of ED and training programs both inside the academy and outside the academy are moving towards that, incorporating more training in data-based decision making, data usage.  IES now funds data academies for teachers and school principals and superintendents, to teach them how to use data and actually I see that as potentially scaling up.  If you get that started, then you have school leaders moving into action research, and in some districts they do that.  Then, it is not that big of a step to expand on research utilization and scale it up that way.  </P> <P>I do think you have to spend some time thinking about how you might restructure the school day, or the school calendar perhaps, to allow more time for school leaders to organize into teams, study their schools, look at the research, make some comparisons.  They do not do that very often.  Some of them do it, but it is not that often.  You have conflicts with school cultures, professional culture of schools, teachers.  I do not think anyone in here has built an intercontinental ballistic missile.  If you have you are in the wrong session.  But you have all been in the fourth grade.  Well, a lot of teachers and principals feel deeply that their context matters.  They know what works best, and you can have issues there with receptivity to what research does.  I think you can more effectively link evaluation to performance. </P> <P>Accessibility and relevance have already been talked about.  You can change your reward structure.  There is no reward for academics generally to publish articles in practitioner journals, or it might actually have an effect.  By and large, we publish things that only people just like us will read.  In terms of research dissemination, that is not an effective model.  And then, I also support more localized research consortiums such as the Consortium on Chicago School Research.  I think if you funded and created a number of those, particularly in larger urban systems, you could really see some impact with research that is relevant and would meet the needs of school leaders.  And that is very narrow and focused.  I think that would be beneficial as well.</P> <P>Frederick M. Hess:  Okay.  Thank you, Lance.  Pat </P> <P>Pascal Forgione:  Thank you.</P> <P>Frederick M.  Hess:  As a district leader.</P> <P>Pascal Forgione:  I guess I m practiced here.  I ve organized my brief comments on these two well-developed papers on the use of educational research by an educational leader, such as my current role as superintendent of a large complex urban district of 82,000 children.  I want to focus on five comments posted on the papers.  But first, you might ask why Austin s perspective should be considered here today.  I believe it is probably because of our recent standing in the release of the Trial Urban District Assessment.  Eleven urban districts took NAEP.  And out of that, I m very proud to say, it was grade 4 and 8 Reading, Math, and Science.  We were tested in 2005, and Austin students, who overall are Hispanic and African-American, scored first or second.  And we are at or above the national average on our score.  So hopefully, these kinds of results together with our state testing results say we must be doing something.  </P> <P>Now the question is: are we doing anything that is really research-based?  And is that leading to this or is it just idiographic behavior of a dictatorial superintendent?</P> <P>So let me turn to the five areas in the paper that I wanted to comment.</P> <P>First, I want to comment on the persistent barriers that Lance brought up to establish a positive nexus between research and practice.  His paper does an excellent job in discussing the complexities of the barriers of using research in the real world of public administration.  I particularly commend you the discussion on page 13 and 19, where he talks about incomplete information in a loosely coupled system.  I thought I was back with Jim March at Stanford.  Indeed, I believe that too often, our predominant conception of research, have a direct impact on practice is too simplistic and too rationalistic.  As Lance states, School districts and schools rarely make decisions utilizing complete information.  Decision making in education often takes place under uncertainty.   </P> <P>Amen.  I fully agree that the world of public school administration and, as Rick pointed I m in my fourth CEOship at the Federal National, State, and now local level, I do believe is defined by incomplete information in loosely coupled systems.</P> <P>The second point emerges from Lance s discussion of schools structured for systematic organizational learning, professional learning communities.  I advocate that he should further enhance this section by adding the importance of two additional features.  I believe that the hype of performing school districts we are now beginning to monitor, are really becoming productive, because of more systematic ways in their decision making and greater use of research in two areas.  </P> <P>First, having a theory of action to guide district wide improvement of instruction and learning.  You have got to have a theory of action.  </P> <P>Second, you have got to establish a common language to describe and guide teaching and learning across the district and its classrooms.  I often refer to Austin, when I came in 1999, as e pluribus-ville no unum.   Austin had no common anything, including negative watch Moody s rating, et cetera, but you have got to build some unum.  And that is where you have to decide where are the starting points and where do you it.</P> <P>Let me use my district and my partnership with the Institute for Learning at the University of Pittsburgh, Professor Lauren Resnick and her team of researchers, as an example of both critical elements.  Having an external agent to support and try your systemic use of research to improve school practices is a necessary but not sufficient condition.  And there is a handout, I do not know if it has been passed out, but I would like to share with you.  </P> <P>Obviously, I think, we all agree in our theory of action that all student learning at high standards is the mantra.  Everyone says it.  But the question is, when you read Michael Fullen, what is your theory of education?  What is the coherent set of ideas concerning curricular and pedagogical practices and organizational behaviors that guide instructional practices?  For example, in Austin, we believe we are going to do it by being more effective in high-quality teaching.  That is our theory of education high quality teaching.  And we have a set of nine principles of learning that Lauren has worked out that are research-based, and that she updates.  And we continue to reflect, organizing for effort, clear expectations, accountable talk, rigorous content and a thinking curriculum.  These principles of learning make this come alive.  This is what we should do in that classroom everyday.  </P> <P>Now the question becomes, what is your theory of action to get you there?  And that is kind of where we have to shift to some strategy.  And this is our theory of change.  Because when I see a lot of school districts, what I see them doing is: you test and you remediate.  That is their theory of action.  That is not acceptable.  We believe that there has got to be a district set of strategies; working with the superintendent and chief academic officer, working with my associate superintendent and my principal, working with my leaders on the campus, the teacher leaders, and the instructional leaders.  Those are the strategies that we have to commit to.  Then, you also have to have a set of activities you are going to focus on that will define and put the elements into your plan. </P> <P>The final thing that I find most districts do not have is a set of design principles.  What are your design principles?  Do you believe in two-way accountability?  Do you believe in nested learning communities?  Do you believe in effort creating intelligence?  Do you believe in parent involvement?  This is where no magic bullet is going to be adopted in Austin unless you can get through this lens.  And this is a good lens that forces us often to reflect on who we are, to make sure we know where we are going.  </P> <P>Again, the partnership with IFL is both our principles of learning.  This is a common language to talk about teaching and learning.  I do not want this [indiscernible].  All of them had their own language.  I have to have some unum in my district to talk about teaching and learning.  And when you go into classrooms, I talk to kids and not teachers.  I ask them to show me the rubric for quality work.  Do I see student work that reflects that?  And so, this is the power of having a common language of teaching and learning.  Standards are essential.  Effort creating intelligence is essential.  </P> <P>But I also believe in having a language of teaching and learning together with the theory of action.  But I also must admit in preparing for this conference, I m beginning to respect what Lauren Resnick has done for us.  We arm wrestled like crazy.  In fact, I pushed the hell out of her because sometimes she is not fast enough on stuff, like low performing schools.  But if it was not for her, having this research objectivity, it really does force us to think more deeply and to raise questions that my own staff would not dare raise to me, but she will.  And I think it its that critical act of friendship as a researcher that perhaps, that intermediary is important. </P> <P>I would now like to shift there is another missing element that I think came through in the paper by Lance.  And it has to do with: we do not have research on implementation.  We act like if you have a reading program that is all you need to know what works.  Well, I will tell you, my district is not uni-dimensional.  How you apply that in different settings?  We need a much bigger investment by IES on program implementation because I think we do not know enough about how to put things in place in a school that has a lot of ELLs versus English learners.  Those are very different contexts.  You would want to approach it differently.  </P> <P>Finally, I would like to comment upon the issue of implementation fidelity.  Recently, we have really begun to realize that consistency of implementation and having models is an asset.  We recently received a very large Gates and Dell grant.  And as we began to talk with them about this grant, we realize that Gates really wanted to assist us in thinking deeply about our redesign.  You see, we do not know enough about high schools yet, folks.  I studied high school for three years, SREB, Stanford, Linda Darling-Hammond, and I still do not have the strategies that are going to work.  I can do elementary, but I cannot do high schools yet.  </P> <P>But we have got to build that knowledge base.  But I must say that in our relationship in the last year with Gates, they have really forced us.  Gates is no longer stuck on the small-school-only model.  They really are willing to commit to the conversion of big, ugly, comprehensive high schools.  And that is what my parents want.  They love their comprehensive high school.  But how do you turn that into small learning communities?  And I must admit that the partnership with Gates had been a terrific learning opportunity for both of us to say, How do you bring rigor, relationship, relevance, and, obviously, results to this community?   </P> <P>But now, to receive the Gates grant, you have to use proven models, or they would not fund you.  So what we ended up doing with Gates, we decided, for example, was that all my schools over the years said they wanted personalization.  The anonymous high school has to bring personalization.  So you want to get adults with kids.  The strategy they all came up with, all 11, are what are called advisories.  Now, in the old days, everyone would have picked their own and built a hybrid.  We said, You either do first things first, advisories, or you do education for social responsibility.  ESR advisory.   That is a change in our behavior.  </P> <P>And I can tell you now, we are learning from each other.  We have got two models to test them out, to share with each other.  And I think it is this kind of fidelity of implementation.  Plus, we have a project management strategy to monitor the implementation, which we have never done before.  We always kick it off and assume it is going to end up there.  It does not go there without a compass and a lot of rigor.  And that is called evaluation.  So I think both IFL and the Gates process have made us a better school district in terms of reflecting on where we are at.  </P> <P> The final comment I want to make was on the Dunn and West paper.  I found it much easier to deal with the courts than the legislature.  And that was the one thing in the paper.  When I got deposed for seven hours, and I was on the stand for seven and a half hours as the plaintiff s first witness in West Orange Cove, I could tell our story.  I could bring the research forward.  In three sessions, I have not had the chance to change the Texas legislature or the deliberation on any policy issue.  It is all done behind closed doors with reference groups, and that is different.  So I ll tell you.  I ll take the courts over the current legislature.  Thank you.</P> <P> Frederick M. Hess:  All right.  Thanks, Pat.  William. </P> <P> William Howell:  There being three discussants on this panel and it being late in the day, I m going to get right to it and provide a couple of points for each of the papers, and then a more general point about the role of the public in these debates.  The public has not appeared in any of the conversations that we have had.  And I think it is there that we can find an antidote to some of the despair, actually, that pervaded some of these conversations.  So I want to say something about audience, something about influence, and something about discipline, in turn.  </P> <P>Audience: this speaks to Lance s paper.  Thinking about what the set of incentives that face a junior faculty member in political science who wants to get tenure and is interested in doing work on education.  First, they have got strong incentives not to work on education and to the extent that they do work on education, their intended audience for the most part consists of a set of reviewers at the most technical journals that they can get their work into and a tenure review committee.  </P> <P>That is who they have in mind when they write their work.  And they have every reason to do that because their career prospects are structured in ways that they are linked not to whether or not they can disseminate their findings broadly, or whether or not all of us in this room or a significant portion of us in this room know about their research.  It is whether or not they can place their research in high-profile academic journals, and whether they can convince a small segment of their colleagues that they are really smart.  </P> <P>And those are the set of incentives that academics face.  And it is great to talk about how they should speak in ways that are clear and that they should spread their word more broadly.  And by reference to some set of criteria, they should.  But the incentives that they face are not to do the kinds of things that I think a lot of us are recommending that they do.  </P> <P>With that in mind, it is not much of a surprise that their research does not make its way directly into schools.  That we do not have reading groups, that is among teachers and principals, that are reading the latest edition of the Quarterly Journal of Economics and trying to make sense of the findings that are in there.  The audience for whom those papers were written was not for school leaders or for teachers.  And I think Lance has done a really good job of walking through lots of other reasons why this research does not have a direct impact on the behavior of teachers or school leaders.  But it is hard to imagine a model of research writing and education practice that would bridge the gap between these two communities.  </P> <P>In a meaningful way, I think that the latter part of your paper, where you outlined some instances where you can get these incentives aligned, is where I think there is a ray of hope.  And so, for instance, one of the things that schools have are data, and things that faculty have a hard time getting hold of.  Research advocates have a hard time getting hold of data.  And so things like the Consortium in Chicago, where a school district comes forward and says, I have this problem.  And I m trying to figure out what the impact of this program is on these people.  And we have all these data, we just do not know how to make sense of them.   </P> <P>I think there are any number of faculty would love to jump at that opportunity, analyze those data, and they can write their technical pieces, right?  At the same time, they can participate in the sort of direct effort to problem solve within a district.  And so those as models, rather than the notion that teachers or school leaders will somehow read undigested wide body of educational research, struck me as more profitable.  So that is audience.</P> <P> Influence:  And this speaks to the Dunn and West paper, which strikes me as the beginnings of what could be really a great book.  There are some rich issues here in play.  They present lots of evidence that education research appears in court cases.  It appears in court cases with rising frequency.  They pay a lot of attention to variations between state and federal courts.  They pay attention to the prevalence of research across different policy domains.  And they also pay attention to the existence of research both at the guilt phase of trials, and then also at the remedy phase.  </P> <P>And what I kept wondering reading their paper is: does the appearance of research translate into the influence of decisions that are actually rendered?  And that is a really tough thing to get a hold on.  But there is reason here for doubt.  There is a vibrant body of scholarship within political science.  I m going to go back to my discipline again looking at the judicial decision making.  What are the sources of judicial and decision making?  What drives judges to rule at the way that they do?  The single best predictor that you have of how a Supreme Court judge is going to rule on the merits of the case is if you got measures of their ideology, if you know how liberal or conservative they are.  </P> <P>And so, there are lots of people out there trying to figure out ways of measuring how liberal or conservative different justices are.  But if you have got that, you have got a whole lot of leverage in trying to figure out what actual decisions are going to be rendered.  And along with that, there is a wide body of research that suggests that legal constraints are not so constraining on the decisions that Supreme Court justices actually render, so notions of precedence, stare decisis, right? </P> <P>It is talked about.  It is prevalent throughout the rulings that are actually rendered, but the evidence is that it acts as a binding constraint.  That is, judges would have ruled in ways that are different from the ways that they would prefer to in the absence of that constraint.  And this is in an area where they actually receive formal training.  Right?  You go to law school to learn the law and to learn what a set of clear legal principles are.  And so if these do not act as constraints, to what extent does academic research, to the extent that it appears, act as a binding constraint?  Are there instances where judges bump up against the body of research that they find really uncomfortable and say, Well, shoot.  I would like to rule this way, but I have to rule this way.   </P> <P>It may be, it may well be.  But I think the challenge is to try to figure out the condition under which, and how much.  But certainly, there is room for skepticism.  </P> <P> Discipline part:  So, what is it that we can hope for with academic research on issues involving education?  My own thinking about this, is that the most that we can hope for is that academic research can discipline the conversations that we have about a variety of education policies.  And when I say the most, I actually think that is quite a lot, to the extent that academic research can structure how we talk about a set of issues.  And vitally, what academic research can plausibly do is set boundaries for what constitutes a reasonable argument among reasonable people.  Right?  So, to what extent can you come out in the face of a wide body of research and come out and make claims on behalf of the things that you are used to making claims for?  The hope is that academic research can discipline that a bit.  </P> <P>And again, among reasonable people, I think it is really easy to point to lots of hysterical folk on both sides of any divide and say, Look.  Academic research cannot possibly make a difference because there are all of these people, who have made up their minds.  And they are hollering back at one another and they are using research in ways that are patently irresponsible.  And they are doing things that the researchers themselves would find infuriating.   Fair enough.  </P> <P>So the point of hope, at least, where I turned to for hope, is to actually look at the American public.  And on the one hand, here too, to bring back political science.  Political science had gotten a lot of leverage out of making fun of Joe Public, how little he knows about politics generally, and how unconstrained his views are, and how foolish he/she can often appear.  </P> <P>But it is also is the case that average citizens, for the most part, are really quite pragmatic, and whereas political elites tend to on the one hand: if you are for choice, then you are really skeptical about any sort of top down accountability system.  Much of the American public are open to both notions.  And it is here, I think, that academic research can have a real impact, constructing the kinds of conversations that they have.  And ultimately, I would like to think that is where these policies debates will actually be resolved.  </P> <P>So when controversial research comes out and interest groups on both sides of the divide start hollering, they are hollering not to try to convince one another.  There is this recognition that actually is something at stake.  And just a part of what is at stake is the understanding of an inattentive and poorly informed public, but a public that is quite pragmatic and open to a wide variety of ideas about what might actually help America s kids.  </P> <P> Frederick M. Hess:  Thank you, William.</P> <P> Warren Simmons:  This is actually a very pertinent conversation for me.  I ll give you a brief and full description of my background.  I started out as a basic researcher, PhD, Department of Psychology from Cornell University, did a post doctorate at Rockefeller University, spent the first three years of my career talking to other researchers and realized that I wanted to change practices, I had to talk to somebody else.  But I quite frankly did not know how to get to other audiences.  </P> <P>And so, I started a long sojourn and took me to the National Institute of Education and to OERI, office of Bilingual Education, the Army Research Institute, then into districts working in Prince George s County, Maryland, under John Murphy as the central office administrator, then to the foundation world at Annie E. Casey Foundation, then to reform support organizations in Philadelphia, and now to the Annenberg Institute at Brown University.  But all those were efforts to take some fundamental research knowledge that I had, that I thought was important, and get it into the hands of people who could make greater use of it.  </P> <P>And in thinking about the papers, I think that all these papers and the conversations of these four have made three fundamental points.  Politics seems to have an inherent part in policy making and indeed, research. It is hard to separate the ideology and values from the nature of research.  The districts struggle to be learning organizations and both districts and researchers struggle to adapt their work so that it can be taken to scale.  </P> <P>Having said that, I have been thinking about whether in fact we are addressing the right question.  Because based on my 30 years of experience, the answer to that question, does education research influence policy or not? is that it does.  The problem is in both the nature of education and the nature of research.  So in part of our conversation, we are using the term education research as if it constitutes a coherent body of knowledge.  It does not.  There is research on a variety of different approaches and there are a variety of different findings within an approach; school restructuring, frequent instruction and assessments, school finance, school choice, multicultural education, bilingual education, left brain-right brain, multiple intelligences.  </P> <P>All of these areas are areas that are reflected in the work of schools and in each of those areas, there is a conflicting body of research evidence that says it works or it does not work, or it works on these circumstances and does not work on others.  So, if that is the nature of research, to me it is not surprising that traditional districts look the way they do.  That is to say, they are organizations that use research.  But they are using research on radical different approaches with conflicting findings and research that is applied to different levels of the system; curriculum, instructions, school restructuring, operation of a region, operation of various central offices with very different functions.  </P> <P>So when I worked in districts, I find that people are either using research or using research that has been translated by other organizations.  But they are not doing so in an aligned, coherent way because, in some fact, the field is not aligned and coherent.  A testimony of that is if you go to a school of education as I did, when I was working in Philadelphia and I took the district standards to the School of Education.  Temple produced 45 percent of our teachers.  And I said, Gee, would it not be nice if you guys changed your approach so that teachers came up prepared to teach to these standards.   And the dean loved it, and the faculty voted it down.  Because in fact, I guess the job of schools of education is not to teach to a particular aligned approach but to teach a variety of different theories and disciplines and approaches and have wonderful arguments.  </P> <P>That is perfect for an educational institution.  It is not so good for an institution that is trying to create alignment and coherence and reach outcomes at scale. And that requirement of education systems is a very recent one.  So, in the last decade or so or more, standards-based reform was the first time that schools and school systems had actually been required, at least by law and by policy, to create coherent aligned systems of instruction to get the vast majority of students to some kind of proficient or advanced levels of performance.  Prior to that, these systems, and many of them still do in the remnants, operated against multiple standards, multiple programs, and multiple tracks which drew on different research disciples to inform their practice.  So that is one problem.  </P> <P>So how did we take an education system that was fragmented, worked off multiple standards, multiple approaches at different levels, and therefore drew on research to inform each of those separate programs?  How do you take a system and get it to use research in a more aligned and coherent way?  Then, how do you get into institutions that generate the research to foster more alignment?  So I do not think you get that by asking this question because I think the answer to this question, given the current nature of research institutions and the current nature of the old nature of districts is, Yes.  Research does influence policy, but it does not influence in ways that get to this other question.   </P> <P>And the question I have is how can you can you design research that generates policy that leads to continuous improvement in outcomes at scale?  I think that is the heart of the question that we are trying to ask or at least, that is my interpretation of the heart of the question that has underlined these papers and underlies that question, but it is not asked specifically.  So I ll repeat that question.  How can we generate research that informs policy that produces practice and outcomes at scale?  And for me, the other piece of this is that the research enterprise, I think we have to accept, is necessarily political and ideological and value driven.  That is the nature of policy, and that is the nature of research.  </P> <P>I think the question before is not necessarily how you insulate oneself from that.   I do not know an advocate of constructivism who every found that constructivism does not work in their findings.  I do not know of an advocate of any particular point, who has a research history that actually counters these kinds of theories that the person supports.  But the question for me is how do you moderate the influence of politics in ideology so that people can generate research at scale?  </P> <P>The way I attempt to do that is actually, I m glad I listened to James Kim s presentation this morning because I felt guilty over the years for responding to invitations from Rick Hess and Chester Finn and those guys who think very differently from me.  But now I realize I do so because of the importance of adversarial collaboration.  That is to say, if we only keep ourselves in rooms of like-minded people, we will never be pushed to look at broader evidence and challenge our basic assumptions and notions.  </P> <P>And so typically, what I see the need for, if we want to address this question of getting district s research that generates improvement at scale, is that there are some propositions that are evidenced based.  They have politics and AEL attached to them that are on the table.  There is a proposition of school choice.  There is a proposition of standards-based instruction.  There is a proposition of restructuring schools and the latest iteration means making them small.  There is a proposition of culturally responsive pedagogy.  There is a proposition of extended learning.  There are other propositions.  They are major propositions that districts grapple with in their work.  And what they rarely get from researches is anything that allows them to integrate these different perspectives into something that they can manage to implement given their current capacity.  </P> <P> I had a colleague when I was part of a district reform, research-based reform initiative.  And the researchers took to the way to infuse research into the district was to bring five expert researchers in different areas; Special Education, Bilingual Education, Curriculum and Instruction, Literacy, Mathematics, to meet with teachers and curriculum specialists to tell them what the best practices were.  And they spent lots of money, brought together hundreds of teachers, and each one of these people set for about an hour each and basically laid out research-based best practices.  </P> <P>As I listened to this over three hours, I was struck by the fact that the educators in the room all thought it was a wonderful experience.  They videotaped it.  But I was struck by the fact that if you did everything these individuals asked you to do, that district s budgets would have to be increased ten-fold.  And schools would fundamentally have to be reorganized because none of them made any effort to accommodate other needs, other approaches, the amount of money available, the expertise of teachers, the structure of the school, the nature of the district, and the politics.  And they were not required to do so.  And so, they brought their research-based practices to the district, laid them out in one hour presentations, very lousy PowerPoint slides, and those researchers do not use PowerPoint very well, and then left it up to the districts to do the integration.  </P> <P>And essentially, what each office did, Bilingual office, Special ED office, they listened to their particular expert and then advocated for those practices to the superintendent and the school board as a whole.  And they had a zero sum game of success.  So, I think that we are moving away from this.  And the way we are moving away from it, I think, is to start developing some criteria that will allow us to enact answers to the questions of, how can education research influence policy in ways that promote large-scale improvement of practice and outcomes?  </P> <P>I think we can generate another set of standards.  I think qualities, issues of rigor and validity, and reliability that were raised earlier are critically important.  I think another standard to judge research by, in this issue of scale, is relevance.  Is the research responsive to key questions driven by the world and needs of practice versus the needs of a particular interest of a particular discipline?  </P> <P>So this issue of relevance, I think, as Gina said earlier on, is this issue of usability feasibility.  Again, those six researchers who all came together and offered best practices that, in my judgment, that district did not have the capacity with its resources and expertise to do at that moment in time they were not making suggestions that were usable and feasible.  And so, how do you generate functions that take important concepts and ideas but convert them into useful policies, practices, tools, and strategies that can be applied under today s conditions while you are pushing for a new and larger system?  </P> <P>And the final piece that I think has been mentioned is the issue of ownership.  That is to say, since this work is very value-laden, we do have some work to do to ensure that decision makers, policymakers, and stakeholders own some of this research.  And the way in which they own it is to some extent by participating.  And so I think that the development of these collaboratives that bring practitioners and researchers together to formulate the questions and help researchers get a better understanding of the context, create these kinds of ownership.  So I think I would offer a new set of standards and, given that new set of standards, I think we need to rethink the policy, practice and research infrastructure.  </P> <P>What I have seen work best at districts that are becoming more evidence-based, is that they have reform support organizations that work alongside them, like the Institute for Learning, that have applied research groups like the Chicago Consortium.  They develop tools and policies, they also pay attention to community engagement and communication, which is one of our traditional weaknesses, both on the research side and education side. </P> <P>And then finally, they pay attention to the need for cross-sector leadership development. Mayors, increasingly, school board members, and corporate leaders need to have this understanding.  It has to be evidence-based, but there has to be some kind of infrastructure at the local level and at the state level that parallels what we now have at the federal level, which  achieves for them what Business Roundtable and other organizations achieved.  I m done.</P> <P> Frederick M. Hess:  Outstanding.  Thank you, Warren.</P> <P>Josh, Lance, anything you all want to toss in before we open it up. </P> <P>Frederick M.  Hess:  Okay.  Let s go ahead and do it.  If you have questions, please catch Julie or catch Morgan floating around.  Okay.  Yes, ma am.</P> <P> Female Voice:  This kind of goes to the idea of the fidelity and implementation.  We have heard that there is a demand for research for the policymakers and that there is just uneven use for the actual practitioners.  So, if ours is a government of the people, by the people, for the people, it seems that there should be some kind of harmony between the policy that is made and crafted and the desires of the populace.  How can we create a greater demand for research from the practitioners?  If practitioners have that appetite for good, sound research?  And then, through doing that, perhaps there may be some kind of harmony between policy and practice.  In kind of along the lines of that, will that improve the fidelity between policy and practice?  And if so, who are the main stakeholders to make that happen?</P> <P> Frederick M. Hess:  Okay.</P> <P> Lance Fusarelli:  Everybody is looking at me so I guess I ll answer.  I think it is possible.  I still think that there are some significant structural barriers and cultural barriers between the culture of school leaders, the culture of practitioners and how they do things.  I do not think it is true that school leaders do not want to use data.  But as several people said before, context will always trump abstract statistics.  A Texas superintendent said, My school board says it is my kids, my policies, my money, and we are going to do it our way.  And we are going to do what we think is best.   </P> <P>And in the amount of personal beliefs and how strongly that local context factors in to decision making, is really significant, and I think sometimes understated.  I think only through collaboration, I think.  And by having researchers have some credibility and being willing to listen to the needs of school districts and what they need research on and the kinds of questions that they are interested in.  I think you can build that, but that requires communication.  And there are too many reward structures on both sides of the realm that are not built to foster that type of communication.</P> <P> Gregory McGinity:  Hi, Gregory McGinity from the Broad Foundation.  I wanted to go to this question about incentives because I think that is really, really, important.  And when I thought about it, if you think about it in a healthcare context, doctors have a pretty strong incentive to use best practices and research in the work that they do.  It is either they want to do the best for their patients, which I think is true of teachers in terms of students.  But there is also medical malpractice.  If they do not do it, there potentially could be a consequence.  </P> <P>And then, looking at the model that we have looked at DARPA, Department of Defense, the incentive for research there is driven by the Department, the consumer.  So, if an academic researcher wants to work on something, they have got to find a partner who is general in the military saying, I have a problem.  I need this to be solved.   And so the question is, in the school context, how do you go about finding a way for the school districts, which I think Pat is right, at the foundation, we think that is the right lever of change, to give the school districts the power to drive some of that research?  </P> <P>And so potentially, is it federal funding that goes to the districts to allow them to have some additional research money to say, We need to solve this problem about eighth grade English Language learners.  We do not know what to do.  There is a whole bunch of different products out there.  How do we know which one will work for our kids?   Or is there some other method where you potentially bring together a coalition of superintendents like Mike Casserly sought to do with Council of Great City Schools, to try and drive that connection?  So anyway, I just put that out for the panel s consideration and any thoughts you might have.</P> <P> Frederick M. Hess:  Okay.  Pat, any thought on that?</P> <P> Pascal Forgione:  I m intrigued by your question because obviously, what has happened over the last two decades, there is no longer any funding sources.  So it has kind of given up on it.  And in fact, that is perhaps why people see me too often because they want me to buy into their problem and their solution.  They do not often come and say, How can I help you?   But I think some of these more thoughtful ways, I think we do not tap teachers enough and teacher associations and unions.</P> <P>We just are doing book studies on Milby, MacGlaughlin and Joan [phonetic] book, Professional Learning Communities, and my teachers are so turned on by it because they really feel this is the key to professionalization, and they can study each other, grow with each other.  But you kind of have to build it slowly and build a culture for that.  It would be great if we had some funding that could allow them to explore some issues they care about.  Unfortunately, I do not find a lot of university people have the skill set that you may want.  And that is why I tend to look around the country and find who is doing the best practice and try to find a way to get them to come to Austin cheaply.  </P> <P> Warren Simmons:  My response to that question is that I think we need to look for new kinds of hybrid organizations to do the research.  In Philadelphia, there was a strong network of teachers who were engaged in action research supported by the University of Pennsylvania, also supported by the National Writing Project, and also linked to the work of Research for Action.  I think having this research function sort of quasi-independent from the school district, buffered it from the ideological shifts and changes that often occur when superintendents leave and often present in the research and evaluation units of the districts themselves.  </P> <P>And so I know that many funders are now in conversations with research groups and cities across the country to think about how to build similar to the Chicago Consortium of the School Research in Chicago, in New York, in Boston, in Philadelphia, and other places.  So I think about action-based research networks, research consortia that bring together researches and practitioners who are problem-centered and applied, and their focus as a way that many funders are thinking about moving.  </P> <P> Frederick M. Hess: I guess, partly this question of the usability of research, I think Warren, as you put it before.  It strikes me that we talk about a couple of different kinds of research here.  At one level, we are talking about research that influences policy.  It is, I think, more akin to kind of what Josh and Marty sketched in their paper.  We are talking about broad changes to statute, to rules at the level of a district or a state that have influenced the way lots of kids are educated.  We are also talking in the conversation though about much more micro-level research, that teachers are doing in their classrooms or that collaboratives are doing partnership with teachers, which is intended kind of as Pat referred to in that context -- trying to make these things work.  </P> <P>I guess, this strikes me very much what Richard Ingersoll spoke about this morning.  That policymakers took his research, and at one level, let s talk about what goes on in terms of the kind of insight that black box of running schools.  But they wanted to use it to solve macro problems in a way that they could get leverage.  Is part of the problem here that we are not doing an effective job of separating out what we mean by evaluation and action research and implementation studies from broader efforts to study research in a way that should and could influence policy?  Or how should we think about those distinctions in a useful way?  William? </P> <P> William Howell:  I ll reiterate something that I said before, and then I think others have recommended as well.  And that is there are opportunities and unexploited opportunities, or underexploited opportunities, to take communities of researchers and communities of practitioners and put them together; not in spite of their incentives but because of them.  </P> <P>And so, again, the thing that researchers, who have to satisfy a handful of their peers, need is, if they are going to study education, its access to schools, and lots of data about the performance of kids.  You give them that.  And you say, This is the thing that I m interested in even if it is for one district.  They can satisfy their needs by them doing really high-quality research and asking them important questions.  And hopefully, it is questions that are being driven by the concerns and needs of a particular district and what they can bring then to the table is a set of quantitative [indiscernible], work with data.  They have lots of them in the area there with issues of causal inference and how you design a study.  That is what they have got.  They are ready to go.  And so, those strike me as some of the most hopeful opportunities to bring these communities together.</P> <P> Frederick M. Hess:  But now that kind of collaboration, those researches are not bringing to the table what Warren and Pat are talking about in terms of attention to fidelity of implementation of who wants attention to context.  What they are bringing is a skill set.  Is that not right?</P> <P> William Howell:  Well, so, but presumably, the question that is being asked is going to be driven by the needs of the district.  The district would come forward and say, We are worrying about this particular program for this group of kids.  We do not know if it is working or not.   That is something which an outsider could come in, and start struggling with the existing data and not provide the definitive answer on it but begin to make inroads.  So that the conversation becomes about this research for this district, not do vouchers work, right?  And now, we all have to get agitated, but then do a particular type of choice for a particular type of district for a particular population, is there evidence of games or losses?  </P> <P> Frederick M. Hess:  Okay.  [inaudible]</P> <P> Gina Burkhardt:  So --</P> <P> Frederick M. Hess:  Gina, would you introduce yourself for latecomers.</P> <P> Gina Burkhardt:  I m Gina Burkhardt from Learning Point Associates.  So Warren and William, to put a little reality, that consortium in Chicago has access to the data, and they do this study on social promotion.  And they do it in conjunction with the district because it is a real issue.  And the results come out and no more social promotion, but the politics trump both the research and the school district need.  How do we get around that?</P> <P> Warren Simmons:  Well, for me, that is where this broader infrastructure is important because I think that is where community engagement comes in.  That is to say, politics are part of this.  But when politics begins to work in ways that belie the evidence, then the only way to turn it around that I know of, is not to have researchers testify before a city council or state legislature or members of congress necessarily, but to have informed advocacy.  </P> <P>And unfortunately, in many cities, the community organizing groups have not had relationships with research applied research organizations that allow them to have informed advocacy.  That is beginning to change in New York. The Institute for Learning, which is now part of the Annenberg Institute, has been working with community organizers.  And they have activated and informed community action against in a way that is more research-based, when the politics moves away from the evidence.  I have seen that happen in Philadelphia as well.  That is why I think community engagement is also an important part of this work.  </P> <P> Frederick M. Hess:  Okay, Joe.  Oh, Mike -- </P> <P> Michael Petrilli:  Mike Petrilli again.  So, was not accountability supposed to change all this.  For Lance or William, why is the incentive of accountability not enough for educators to change their behaviors, and say, Wow.  We have to get these test scores up.  We are going to search the research as much as possible. </P> <P> Warren Simmons:  I can answer that question.  It is enough of an incident to get them to change their behavior.  It is not enough of an incentive to get them to use research to lead to continuous improvement.  That is to say if you have to on a year-by-year basis improve some test scores, you will try different things.  And the different things that you are now trying are labeled research-based.  So that in fact, in Philadelphia, remember the comprehensive school reform designs that were all supposed to be research-based, started with New American Schools that were eleven, and then somehow, it grew to forty-four.  </P> <P>Well, schools started adopting research-based models, whether those models fit their conditions, their populations or not, because they were research-based so we had enough of an incentive to get people to adopt things that were research-based.  We did not have incentives enough or the quality of research enough to leave them to ask the question, Do these models fit my conditions and my needs?   So I think the question becomes, how do we increase the incentives so that research becomes more conditionalized and fosters continuous improvement?</P> <P> Frederick M. Hess:  Although, but Lance s [indiscernible] jump on top of it because I thought I heard you saying that not only were they not being able to make the distinction that Warren just referenced, the right fit for their school, but it sounds like in many cases, they are not being nearly as proactive as the accountability folks suggested.  Right, the suggestion was accountability gets folks focused on the bottom line.  They will go out.  They will seek ways to improve and steer an achievement. And it sounds like from the story you told, that there is not nearly as much of that happening six years in the No Child Left Behind era as some have hoped.</P> <P> Lance Fusarelli:  I think there is some of it going on.  But I also do not think that I think you can encourage people to change accountability efforts, things like that.  But you cannot force people to change.  And people underestimate how difficult and how resistant long-standing organizations like schools can be to change.  You have to get first, you have to move people into the mindset where they are ready to accept change.  You cannot just sort of mandate it and say, We must change, because that is where we are going to get resistance. </P> <P>It takes time.  The scale I was talking about, how long it has taken him to change those mindsets.  But that has to be a consistent multilevel pressure and encouragement.  It cannot just be a stick.  It has to be a support.  </P> <P> Frederick M. Hess:  Oh, Pat.  Please.</P> <P> Pascal Forgione:  You triggered in my mind this whole issue of accountability.  I think we have got to realize, there are a set of strategies that you can kick butt and you can make progress.  But when you are talking about 2014, and you are talking about that level productivity, you need a different level strategy.  </P> <P>So what I m finding is: how do you create the second level strategy?  For example, I have done all that basic management instruction stuff, got it fine-tuned.  But I realized my greatest demand was to have one curriculum in my district.  I m going to have a special ed bilingual and regular curriculum.  It is going to be one curriculum.  That to me, I think, is going to be the most powerful innovation that can do well in the next three years.  Can I get all those working together?  So they see one curriculum, and the kids need Math versus the silos.  This is the kind of structure -- I do not know if the federal government, it is certainly not putting enough resources out there to help me even think about it.  And the test scores forced me into kind of a triage because I do not have the systemic at the high school redesign.  </P> <P>I m really trying to do real redesign and trying to meet the needs of 600 kids who have not passed one of four tests.  It is hard to do both.  And so, I just think we do need your policy leadership.  And I think our colleague from Senator Kennedy s office mentioned it today.  What would that improvement framework be that might change the paradigm from shame and blame to development, capacity building? Because this really takes quality teachers as Ingersoll said.  But it is also have to get them in the right places, and you have to keep them and retain them.  </P> <P>And so, this is the complexity of it.  So I really think we are at a stage of reform that I m pleased to be there.  I think we are really going to deal with some very difficult issue.  I do not know if I ll make it.  But the question is, that is not important.  It is, can you really move a system in a systematic, and as Warren said, the alignment, the coherence?  Because everyday some mouse comes out of nowhere that wants to distract me.  But it has got a price tag, and should I do it because of a board member.  And I say, no.  How many times can you say no?  This is the challenge of democracy as you pointed out in both of your papers.  </P> <P> Frederick M. Hess:  Pat, I would love to hear you elaborate a bit.  In what sense is the curricular redesign research-based?  Because there is nothing that actually -- there is nothing by the NCLB metrics, for instance, scientifically-based, shows that the curriculum you guys are moving towards is the most effective.  So as a leader who has been through this stuff, wearing a bunch of hats, what is the process and how do you think about it in terms of this research debates?</P> <P> Pascal Forgione:  It is about vision.  You have to get leaders who have a vision.  And I keep underestimating the power of that vision because that can lead people to do what is right.  But you have to empower them to own the problem.  And so, you are right.  I guess, I do not have a research proof that what I just said will take me there.  But my whole team feels it.  And I can tell my ELL kids need a core rigorous curriculum, but it cannot be a stand-alone.  And I have to have that English is the goal.  </P> <P>So, you are absolutely right.  I guess I m just trying to figure out where is the next strength I could build, that would build coherence and would build kind of capacity building.  And those are two things that I look to.  If I can build capacity and coherence, I have to be on the side of the righteous because many superintendents can do things that can destroy a district overnight.  You do not realize the messages you send and the poor decision you make that has confounding effects on everything else.  </P> <P> Frederick M. Hess:  Okay.  Yes, ma am.</P> <P> Peggy Siegel:  I m Peggy Siegel with Advance ED.  I wanted to pick up on a little bit about what Warren said in the context of what Pat just mentioned.  It would seem that there is a real opportunity here with the No Child Left Behind, where National Governors Association, Council of Chief State School Officers, and National Association State Boards of Education have gone in uniformly and asked for more state flexibility for innovation to be accountable to provide data and research on what is working in real time at the state level collaboratively and to federal policymakers.  </P> <P>So I wanted to, if you really believe in sustainability and continuous improvement and you need the structures to do that, and Pat, the lessons you learned from developing in a very highly-active local school community that you have actually sustained progress over time, are there ways that we can use that as a leverage point to bring the research to the state local collaboration and then, flow it back up from the grassroots to federal policymaker?  So we are not so sub-optimized by programs, time, projects, research driven, but it really is systems driven from the grassroots on up.  </P> <P> Warren Simmons:  I think that this work is just in some places getting underway.  I think that the work that Catherine Snow and her colleagues are doing around literacy supported, I think, in part by the National Science Foundation, being piloted in Boston and other places, and the work of the Data Center is doing in Mathematics under Uri Treisman, is sort of bringing together researchers and developers to develop curriculum materials that are situated against the needs of districts.  I think that is just getting on the way.  </P> <P>I would think and hope that the federal policymakers and researchers would take a look at that work and perhaps make that something that they invest in when No Child Left Behind is reauthorized.  But I think it requires us to have a new theory of action about the research policy practice infrastructure in this country because I think it is unwieldy, necessarily so, because of our loosely coupled system.  </P> <P>But if we are moving toward an alliance set of outcomes that which find it to reach, but I think the question becomes how do we tighten it.  But also, how do we design it so that it is informed well from the bottom up than it is from the top down?  I think that one of the things I will always regret as part of the new standards project is that we invested heavily into the development of new performance-based assessment and did not invest as heavily in the network of teachers who initially were piloting those assessments; and in fact, when the assessments failed, political opposition and technical inferiority perhaps were witnesses.  </P> <P>What has been lasting and what I encountered in places that I go is the network of teachers who continue to inform reform efforts in many of the cities.  And so again, it is an important example for me in my work of how the power of teaching networks is linked with researchers.  </P> <P>But also importantly, I think we do not recognize that researchers and practitioners are necessary but not sufficient.  We have to have people with development expertise.  And then, when we decide that we are going to take success for all the scale, you basically have a group of researchers at Johns Hopkins who really did not have skills that they needed to take that out to scale.  And this was in the standards, did not have the capacity to develop and market a tested scale, and therefore, had to partner with testing companies eventually.  So I think the development community, which is sort of lays out there on its own, developing products again, has to be brought into this conversation as part of this in the infrastructure.</P> <P> Pascal Forgione:  Quick word.  I like your term flexibility for innovation because I think that is going to be key for how we build that.  I find all these accountability mechanisms are sending mechanics to me.  And these mechanics are sending the wrong signals to my teachers.  They are into this mechanistic way to improve instruction, and we are trying to think more systematically.  This is probably a point.  I look for leverage points, I guess.  And perhaps, one of the leverage points you have in front of you.  We are all talking about four by four, right?  But we do not know what that 4th level Math Course could be.  </P> <P>We also would like to have national standards to many people.  So instead of trying to do all of it, couldn t we just together develop a 4th year Math Course that we all agree?  It cannot be Calculus.  It has got to be something that would be good for kids with probability and statistics in it probably, because that is more useful for most of my students.  But I think that could be an opportunity to leverage some good thinking around the whole issue of We do one standard, yet we cannot quite force the system.   But if you put a capstone there, I can allow my system to it, so you do not have to do everything.  </P> <P>But right now, I m building my own capstone course.  I m trying to work with other districts, and it cannot work with the state.  I just feel like I m out there, where we could be more coherent about that, and it is that whole issue of flexibility for innovation.  I think you are right if we had a way to have a grant to work together and a leverage point.  In that way, the public would build some, I think, faith in us that we are value-adding the system.</P> <P> Frederick M. Hess:  Okay, with that let s make that the last word on that panel.  Let s break for ten.  And I would ask you, as we get ready for the last panel, we will actually have a reception right after the last panel so that you can resume your conversations in a less hurried environment.</P> <P> </P> <P>[Start of Panel 5:  Changing the Incentives for Researchers and Decision-Makers]</P> <P> </P> <P>Frederick M. Hess:  Okay, if folks could go ahead and take their seats, please, we will go ahead and get started.  Okay, this is our concluding panel.  There is always that funny tension in a day-long conference where the concluding panel is a place of honor, but that is because you have lost half the people along the march and it is funny how these things work out.  I would like to make sure you all know after we conclude this panel there will be a reception out in the foyer.  Please, you are welcome to join us; our way of giving folks a chance to carry on some of the conversations in a more collegial environment without all the rushing to and fro during the day.  </P> <P>This final panel is going to see if we can talk a little bit about the incentives that operate on the two-halves of the equation that we have been talking about all day.  What are the incentives and structures that work upon the researchers and the research community, and what are the incentives, institutional structures that influence the decisions by policymakers?  We have two papers and then we have three discussants with us to share their thoughts and reactions whom I think they are probably three of the best [indiscernible] people in the country for this conversation.</P> <P>Presenting first is Dan Goldhaber.  Many of you know Dan s work.  Dan is a research associate professor at the University of Washington s Evans School of Public Affairs and an affiliated scholar with the Urban Institute s Education Policy Center.  Dan s work focuses on issues of educational productivity and reform at the K 12 level and the relationship between teacher labor markets and teacher quality.  Dan s co-author on this paper is Dom Brewer of USC.  Speaking second is Ken Wong.  Ken is the Walter and Leonore Annenberg Chair for Education Policy and the director of the Urban Education Policy Program at Brown University and is, I guess, weeks from being arm-twisted into taking over as chair.  Sorry about that.  Better you than me, Ken.  He is also a Professor of Education, Political Science, And Public Policy at Brown.  Ken was the founding director of the federally funded National Center on School Choice, Competition, and Student Achievement.  </P> <P>We then have three discussants with us.  Speaking first will be Russ Whitehurst.  Russ has, of course, been mentioned in absentia more than once today.  Almost all [indiscernible], though, Russ, so this is your crowd. </P> <P>Male Voice:  [Inaudible]</P> <P>Frederick M. Hess:  I could sympathize.  I have that moment.  I call it the Wendy Kopp  moment.  Russ was appointed in 2002 to a six-year term as the first director of the Institute of Education Sciences, the research arm of the US Department of Education.  Russ previously served as US Assistant Secretary for Educational Research and Improvement.  </P> <P>Next, we have Mike McPherson.  Mike is the fifth president of the Spencer Foundation.  Prior to joining the foundation in 2003, he served as President of Macalester College in St. Paul, Minnesota for seven years.  Prior to that, Mike spent twenty-two years as a professor of economics, chair of the Economics Department, and dean of the faculty at Williams College.  </P> <P>And, finally, we have Kathy McCartney.  Kathy is dean of the Harvard Graduate School of Education and the Gerald S. Lesser Professor in Early Childhood Development at Harvard University.  For the past 16 years, she has served as a principal investigator on the National Institute of Child Heath and Human Development s Study of Early Child Care and Youth Development.  Dan, would you please take it away?</P> <P>Dan Goldhaber:  Yes.  Rick, I want to thank you for the invitation to do the paper and to be here.  It is kind of a pleasure to work on a paper that is impressionistic and not burdened by any numbers or regression results.  So, I want to emphasize the caveat - is impressionistic.</P> <P>Frederick M. Hess:  Well, Dan, and your paper actually tells the genesis of this conference, which was about a year ago at Brookings.</P> <P>Dan Goldhaber:  I actually was not sure if that was correct, but that is the genesis.</P> <P>Frederick M. Hess:  That was the genesis.</P> <P>Dan Goldhaber:  So, actually, that is kind of a nice way to lead in.  Rick and Dominic and, maybe, Jeff Henig and a few others were sitting around the table at a conference at Brookings that was about class and school size.  And so, two or three days of conference discussing the research on class and school size.  And probably in consternation hey, is my time being taking out for this - in consternation, the questions was raised:  Why do we have a conference on class size and school size, particularly, class size?  Is that not research that has been done ad nauseum?  Do we not know the answer to the impacts of class size?  And out of that comment, I think, came the idea of what is driving research.  And so that was, at least, the gist of our paper and, I guess, the genesis of the conference.</P> <P>So, let me tell you that my impressionistic and Dominic s impressionistic opinion of where we are with research today -- and that is that it is probably better than it has ever been, but it is still the worst of times.  I think that there is -- if you look at numbers, at the amount of research that has been generated today -- there is probably more than has ever been generated.  There are more research outlets than there ever have been before.  But the evidentiary basis on a lot of questions that our policy interest is still pretty thin and I would say that research is oftentimes misapplied, class size being a very good example of the misapplication of research where we actually know quite a lot about the impacts of that policy intervention.</P> <P>I m going to just drop into saying a little bit about my impressions from having been on a school board, too.  And having been on a school board, I would say that anecdotes really rule the day and that has been raised as an issue a couple of times. But you really need to bring research to another level, I think, for it to overwhelm the power of anecdotes.</P> <P>The other thing that I think it is important is that the incentives - which is what Dominic and I focused on in our paper - for doing research are often not very well-aligned to get research done in a timely way on policy-relevant topics and in a high-quality fashion.  So, we basically take this, as economists, from an economic standpoint.  We do it to an economic model and think about what are the incentives that drive the supply side of research, what are the incentives that drive the demand side of research.  And I can already tell that I'm rushing through my slides.  </P> <P>So, let me just very quickly say the research is supplied by individuals that are operating in a lot of different institutional contexts.  So, you have people at soft-money institutions like the Urban Institute, like RAND, that are driven by issues on a very fundamental level of job security.  So, they are going to be responding to RFPs.  </P> <P>And I think that for the most part responding to RFPs that that kind of work is going to be very policy-relevant because of the nature of the work.  But you really do not have, for the most part, the freedom to kind of think outside the box, except for a few individuals who may have the reputation that allows them to do this, to go and say, I think that you should fund this major study that looks at a particular issue that is sort of not a status quo issue.   It is hard to have few people have the juice to be able to that, operating in those kind of environments.</P> <P>In a hard-money institution, you have got more of that kind of flexibility but, oftentimes, those institutions are ideologically driven.  I do not know whether that influences the individuals or whether you get individuals who are aligned with that institution s thinking but you get research that is, oftentimes, of a particular flavor.  Most of the research that is done in education is still done at universities and most of it is done at universities in colleges of education.  And Dominic and I think that the way that universities are currently structured, this is somewhat problematic.  And it is problematic in terms of the research that is done in colleges of education because, for the most part, people in the college of education do not have the methodological training to address a lot of the issues that come up around non-experimental and/or experimental research.</P> <P>Outside of Ed schools, oftentimes, people have better training but they do not really understand or appreciate cultures or institutions.  Now, these are generalities.  There are a lot of ed schools.  There are some ed schools and some ed school researchers who certainly have the capacity to do good research; Susanna Loeb is a good example of one.  And I think that there are some political scientists, sociologists, economists that are doing good work on education who do appreciate the institutions that are important in interpreting the findings that you get.</P> <P>But and this is a very important point at universities, most of the incentives that drive individuals are not around what is sort of policy-relevant, important to those people that are on the ground in schools, policymakers in states and localities.  The research is being done, as was mentioned earlier, I think, by William, primarily, because you are trying to get tenure.  And that means that you are producing it for a very different set of individuals than those who might be consuming it for the policy purposes that we think are important, ultimately.  </P> <P>It also means, incidentally, that a lot of the people that are supplying research are also consuming research, which is very different than most private sector markets and there are some interesting implications of that.  Most of the and these are just sort of general critiques of what is going on but it particularly applies at universities. </P> <P>The overall level of funding for research is relatively low.  The funding is split across mostly agencies.  The funding that comes out of the federal government, oftentimes and, I think, as a complement to IES and Russ s efforts - has gone in the direction of funding larger studies where you can make causal inferences.  But in my mind, it is an open question as to what happens with the Department of Education if you have a change in administrations.  I really do not think we have a test of IES until we see what happens midway through, particularly, a Democratic administration.</P> <P>Another thing that is important about the research and those who generate it is that folks really benefit from publicity.  And oftentimes, publicity -- what garners publicity is in contrast with what might result in the highest-quality work.  I can say I feel this stress.  So you want research to be out in a timely way but you really do worry about somebody else trumping your research.  </P> <P>And so, the kind of quality checks that one might hope happen and probably have been done in the past where you run it by colleagues, where work gets published in credible journals -- I think it has become harder and harder to have those kind of quality checks in an environment where the internet can be used to push research out, where reporters can go and get research from many, many different outlets.  And it is not just looking at a select number of journals to figure out sort of what is going on in the field and what is credible.  And so, there is this tension between especially for those who are interested in the publicity from the perspective of garnering soft money - getting research out early to get the publicity and getting good quality checks on the work to make sure that the findings are robust.  </P> <P>I will also say that anyone who is an honest empirical researcher will say that you run enough statistical models that you can probably find a model that supports any particular position out there.  And I think the good researchers, they do not do that; this one does not.  But it means that if you are trying to find a result that sort of comports to a nice sound byte for the media, it can be done.  And I'm not up here to name names but in the community, we know that that happens and it is unfortunate and there are some striking examples where research will be put out by someone in the department and somebody else in the department has not even seen the research before it hits the press.  And it is astounding; it is astounding.</P> <P>All right, so what do we do about all these things?  Well, let me tell you what we do.  I think that there are -- first of all, some of what we do is being done.  I think that the What Works Clearinghouse and the new design of the research review panels at IES are all for the good.  But we could do more by way of quality control checks; and we can do more by a way of quality control checks on the front-end of research, i.e., funders.  And I'm thinking particularly of foundations that do more to say, These are our standards for evidence.  This is what we are going to expect out of your research.  This is the kind of review that you are going to go through.   And certainly not all funders have high quality standards; some do, some do not.  </P> <P>I also think that there could be more quality checks on the back-end of research and I really like the idea of having a flagship journal in education; I think that that would be very important.  In economics - again, I'm an economist - there is a very clear hierarchy of journals.  And I have to be honest and say that some of the work that I do not think would get published in an economics journal is -- I send it to educational journals.  And I say that a little bit to be provocative but, also, to tell you that that is the way I think that is the way a lot of economists look at the situation.  </P> <P>So, on the one hand I would argue for a more quality control checks; and then, on the other hand, I would argue for - and this may seem a little bit far a-field - but I would argue for national tests.  And the argument for national tests and I do not think this made it into the final version of our paper because we had to cut it way back is that localities do not really have the capacity, i.e., the end-users of research do not have the capacity on their own to fund the kind of studies that would really show results in a credible causal way.  </P> <P>And so, you really want some banding together -- coalitions of localities to fund the kind of large-scale studies that would be credible.  And right now, when you have disagreement over the value of different state tests, there is not as much incentive for localities to band together and spend their money in ways that might yield these credible results.  So, on the one hand, national tests because it might encourage coalitions, and then on the other hand, national tests because there is a lot of disagreement about the outcome -- not the outcome of the research; the variable of outcome.  </P> <P>It is amazing when you talk to policymakers how much pushback you get about tests.  And I'm not psychometrician, so I do not know I sort of know about the studies, but I do not know in a firsthand way how valid that is.  But I think that having a good national test would put a stand to quality on the outcome of interest and that you would probably get less pushback from folks about the research that is done where that is the outcome of interest.</P> <P>Frederick M. Hess:  Thank you, Dan.  Ken?</P> <P>Kenneth Wong:  Thanks very much, Rick, for organizing this very exciting day of conference.  I think you also provided each one of us with some extensive comments and I found out that I'm not the only one who got five pages of comments on my earlier draft of the paper and I subsequently took it very, very seriously and it has been very, very constructive in revising my paper.  And it sounds like every one of us has undergone more or less the same process.  So, I think this book is going to be very exciting.</P> <P>Talking about the gap between the research community and policymaking institutions, there is nothing new.  This challenge of two cultures has actually been discussed in a book by C.P. Snow and some of you might have read it.  And in that book he talked about science and government as two intellectual moral temperaments.  And Snow then went on to talk about two cultures and I just quote his writings here: To be any good, in his youth at least, a scientist has to think of one thing deeply and obsessively for a long time.  And the administrator -- that means the administrator working in the government setting has to think of a great many things widely in their interconnections for short period of time.   </P> <P>And we have gone through some of these ideas, and Lorraine talked about the short cycle of the electoral process as well as William s notion that there is the imperative of running in the election.  So, if you look at the political side of this incentive nexus between the research and the policymaking interconnection, we need to take very, very serious consideration about the nature of the political aspect of this relationship.  And so, in this paper, since Dan and Dom have already talked about the incentive side on the researcher side I'm going to focus more on the political side; that is, what are the incentives and what are the disincentives for the political leaders to actually think about using research in driving their decision making?  </P> <P>And I make the argument that researchers too often pay little attention to the political dynamics in the policymaking process; and I also argue that the electoral interest, distribution of power and partisan agenda must be taken into full consideration when we talk about the connection between the research and the policymaking communities.  We start off with Political Science 101:  Everybody knows that politicians are self-interest-motivated.  And by that, I mean, that political leaders in their own mind are thinking about the next election; they want to get reelected.  And in order for us researchers to be able to communicate with them more effectively and be able to get their attention and be able to get them to think in our terms, in terms of framing the policy debate, we need to really pay attention to the rationale that drives the decisions and the use of time and allocation of capital as well as their selection of information and that is the concern about the next election.  And there has been a lot of literature in political science that supported this claim.</P> <P>The second feature that I want to emphasize is that on one side you have got individual self-interest of course, they want to win reelection.  But at the same time, they are also representing their constituencies.  They are part of a long tradition of policymaking institution.  They, oftentimes, have a very strong sense of institutional mission.  And that is something that we have not really talked about up to this point; that is, we need to be thinking about how political leaders also think about their political legacy.  </P> <P>So, for example, when President Bush talked about No Child Left Behind, he oftentimes framed it in terms of the political and policy legacy of [indiscernible] reform.  And when you talk about Civil Rights leaders in Congress, like the Black Caucus and so on, they do talk about the fact that they are here representing a larger, broader community that goes beyond the immediate electorate that voted for them.  </P> <P>They also talk about the needs to really create equal educational opportunity across the whole country, not just for their own immediate constituency.  So, there is tradeoff between individual self-interest of winning reelection and also the larger context of making a difference, to leave a policy legacy and to reframe and reshape and broaden the pie, so to speak, and improve quality of life for everybody.  So, I see in the literature that these two are actually tradeoffs and, sometimes, they work together; sometimes they do not.  And what further complicates would be the fact that we have a federal system of government.  </P> <P>So, in other words, political leaders do not behave the same at all three levels of our governmental system.  So, federal level, the elected officials behave in a different way because they have a broader tax base, for one thing; they have federal income tax; they have corporate income tax, and they have a broader constituency.  </P> <P>Whereas at the local level, local governments are primarily based on local property tax and, oftentimes, they have to deal with the reality that people can vote with their feet and they can exit the community and businesses can pull their capital out of the community.  So, whatever policy that they are thinking about has to be respectful of those structural constraints, and these constraints are not going to be easily altered by individuals, regardless of how charismatic the leaders are; they cannot change those constraints.  So, in other words, division of function, division of responsibility and resources mediate some of what we see and how we define politics at different levels of the government.  </P> <P>And one final piece that I want to add is this notion of laboratories of policy experimentation.  Given the fact that we have 14,000 school systems for 50 states plus DC and so on, there has been a long literature, a rich literature, that suggests that state localities are actually sources of policy experimentation.  And some of what we see in No Child Left Behind were actually implemented in Chicago and Houston and some other places before; for example, high stakes testing, termination of social promotion, charter schools, and school choice.  All of these were actually found, prior to NCLB, in Chicago as well as in Houston and some other places.  </P> <P>So, the question here is that when some of these programs were found to be successful, they had a tendency to get federalized.   So, in other words, taking into consideration all these political and institutional features, it is very, very likely to see that there are weak incentives to use research primarily to drive policymaking because we are in a liberal democracy and deliberation is a very costly and time-consuming process and research only features as one of the conditions.  </P> <P>Timing is everything; we know that.  The electoral cycle is short two years or four-years, whereas research sometimes takes six to ten years.  And we talked about that earlier in the other panel.  </P> <P>Decentralized system of education is tough.  You need to get approval to get access.  You also meet the challenge of generalization what you learn in one district may not be applicable to 13,999 districts.  So, what works in one state may not work in other states because of the complexity of the context.</P> <P>Intra-organizational politics is tough.  I think Andrew talked about that in the first paper that there has been, every five years, every 10 years, you see reorganization at the federal level.  And every time you reorganize, you loose about six to eight months of time in terms of organizing and research and [indiscernible].  In [indiscernible] of the status quo, well, there have been strong, well-entrenched interest groups at all levels of our federal system.  </P> <P>So, whatever new ideas that you want to promote first of all, Russ has been trying to get more researchers to be interested in experimental design.  And, well, researchers like us also are well-organized and we try to resist that a little bit.  Not to mention some other well-entrenched interest groups.  So, a status quo politics is one of the factors that will resist some of the innovative initiatives.  </P> <P>But then there are other situations where researchers would gain policy attention; in my paper I talked about two examples.  One is the Koret Task Force based at the Hoover Institution.  The other is the Harvard Civil Rights Project.  Both of them have attracted a lot of policy attention.  The fact that they were very successful is because they were able to bring former federal assistant secretaries as part of the task force, and they also had them design and think about the end-product of that research.  The other is the policy window; when there are uncertainties among policymakers, it opens up policy windows for researchers to play a pivotal role.  And the example is A Nation at Risk, and there are researchers that produced 40 papers submitted to the Nation at Risk National Commission on Excellence.  And they were able to make use of that and, therefore, produced a new paradigm that is [indiscernible] and push us to think about outcomes and efficiency in addition to equity.  </P> <P>And so, there are, of course, investments in marketing research that we all talked about earlier and the governmental actor [indiscernible] is very important.  For example, the Tennessee class size reduction involved the Tennessee governor as well as the state legislature.  And one last point about the importance of new policy attention is given to a demand-side politics, and that is that the consumers are giving us some pressure that we really need to hold schools accountable.</P> <P>And so, skip to the conclusion.  I know that I'm running out of time here.  So what I have here is:  How do we go beyond the two cultures?  And what I like to say is that we need, of course, to enhance the rigor and credibility of research.  We want to separate the advocacy claims from research-based evidence; testing some rival hypothesis, I think, is very important.  We need to mediate this very, very tough challenge.  There is so much divisiveness among ourselves within the research community in terms of methodological approach and substantive initiative that we really need to mediate among ourselves.  But, more importantly, I think we need to learn from other policy arenas.  </P> <P>And here, I would like to propose that maybe we need to think about the National Institutes of Health approach and that is to have congressionally approved categorical institutes that identify strategic agenda that Congress can actually hold accountable on the mission as well as the product of this institute; for example, the Institute for Cancer Research.  And Congress can actually take ownership on the direction and the quality of this work.  And I know that Russ and Michael commented extensively on this suggestion in my paper.  But having said that, that would be my key proposal. </P> <P>Frederick M. Hess:  Okay.  Thank you, Ken.  Russ?</P> <P>Grover J. (Russ) Whitehurst:  Thanks for having the event and I appreciated the two papers.  Both of these wonderful scholars are supported by us, so how could they not have good things to say?  And, actually, there is very little that I read or heard that I would disagree with except the last point, which is Ken s point about categorical institutes.</P> <P>Of course, OERI was organized around five categorical institutes, and did not work.  At one point in the history of OERI, the number of R & D centers ballooned from 11 to 23, in part, based on the logic that Congress wanted to focus research activities where it could have some ability to dictate the focus as well as the funding.  And over that time period, when the centers doubled, the budget of OERI decreased by 21 percent.  I think if you will talk with people of NIH about how they feel about Congress deciding that a particular amount of money will go to the National Cancer Institute, whether that is the place where the research opportunities lie or not, and the sense of the research community, you would find a lot of folks who do not like that model at the NIH.  So, I think we need change.</P> <P>But I think going back to categorical divisions of the research enterprise is probably giving too much role to the political side of these two cultures with respect to what research agendas ought to be.  I could not help and looking at the very important topic of the conference I'm thinking about the world of policymakers and the world of knowledge makers, too I was thinking about it this weekend. I was sitting on my porch and I was watching the squirrels play and both of these communities are squirrelly, if you will.  And if you have ever seen squirrels play, they touch and they fly off in different directions.  The contact is chaotic and it is hard to make a lot sense of it, yet squirrels need it, apparently.  And I think the policymakers and knowledge makers need each other as well.</P> <P>I think a principal point of Dan s paper and I think it is a fundamental point is that there needs to be a market for education research.  But I think one has to go back and say there needs to be a market for education performance.  Up until a few years ago, it was very difficult for a parent or a state education agency, or those appropriating and using funds, to know what to expect in terms of the performance of schools.  </P> <P>Yes, there are general reputational factors and one could look at the tax rate at a particular district.  But what to expect of outcomes was uncertain, indeed.  I think before you have a market, you have to have expectations for performance.  And so, those expectations come from standards, which can be formal standards or just the kind of expectations you have when you enter a restaurant after a lot of restaurant experience, and what should it look like?  Should it be clean?  How should the service be?  There has to be a clear set of expectations for outcomes.  </P> <P>And connected with the expectations, there has to be some form of accountability.  If we run to the stock market as an analogy, there clearly are expectations.  The accountability is that people sell stocks at particular targets are not made for growth.  We need accountability and education.  It can take the form of No Child Left Behind sanctions; it can take the form of choice.  But there has to be a set of expectations.  There has to be an accountability system, loose or formal, for not meeting those expectations.  And [indiscernible] there has to be reliable information.  </P> <P>I have been thinking a lot lately about the post-secondary sector and how it is that we have the most expensive higher education system in the world, and a mediocre one in terms of most measures of international performance.  I think one reason is a complete lack of transparent information on the performance of higher education institutions.  Is Brown University better than the University of Washington?  How so?  How do we know if we were making choices?</P> <P>I think with those things in place a market for education performance that is formed by standards, accountability, and good information there starts to be a reason for the knowledge-maker community to connect with the policymaker community on more than a sporadic squirrelly basis because people want to improve relative performance.  And if the knowledge maker community is producing rigorous research that is also relevant, it will be a useful tool in increasing performance and will find its own market.</P> <P>The Institute of Education Science, of course, has a role to play on this and so let me just tell you as simply as I can what I think that IES has been about and I hope will continue to be about.  One is building a strong federal research agency that can fund rigorous and relevant research.  It is easier to fund rigorous research than it is to fund relevant research, though neither are easy.  So, we put in place a series of processes that I think succeed in funding rigorous research.  We are constantly pushing the research community, however, to engage in the kind of research that Dan was talking about that actually addresses the issues that practitioners and policy have to face as they are struggling to educate kids and make their mark under various accountability systems.  </P> <P>We have to build capacity.  That is a matter of both funding and people.  One of the papers mentioned the relative or, perhaps, both papers mentioned the relatively low level of funding for education research; it is embarrassing.  You go to health and human services, 42% of the discretionary budget is invested in the National Institutes of Health.  And, of course, there are other research entities housed under HHS, as well.  You go to the U.S. Department of Education, less than one percent invested in research.  So there has to be more money.  The Aspen Commission call for doubling of the budget of IES, I think that is a good place to start and I hope that Congress will take that advice their advice seriously.  </P> <P>And we have to build our capacity on the personnel side as well.  We have invested heavily on what we call our Baby Doc Program, which are inter-disciplinary training programs for doctoral students in the educational sciences; about a 160 students in the pipeline.  We hope to have trained over 300 over the next few years and I think that will be quite important in increasing the supply of rigorous research and, I hope, relevant research as well.  If you got rigorous and relevant research, you have to have tools that make it easier for practitioners and policymakers to access that research.  The What Works Clearinghouse is part of that; we have others in the planning stage.  </P> <P>You have to be helpful.  I think there is and none of the papers I have read perhaps it was covered earlier today and I'm sorry I was not here.  In these papers, however, there was no one touched on the personal relationship and how individuals at the right place and the right time can have a significant influence on the utilization of research.  And I think all of us have a responsibility I do, in particular to try to be helpful to policymakers who are struggling to find evidence that is relevant to the decisions they have to make.  I think you have to be political without being politicized in the process.</P> <P>And then, finally, I think a theme that both papers have touched on, and it is an extremely important one, is continuity.  Just as one could imagine the two squirrels, two cultures of squirrels policymakers and knowledge makers one would also have to say that the history of federal government funding and education agency within the federal umbrella is squirrelly as well.  It has jumped around quite a bit; as soon as you think you have got something, Congress changes it again.  We cannot expect people to make careers in education research unless they can anticipate that there will be a stable federal structure to support that research.  </P> <P>So, we all need to stick to it, including Congress as it considers the reauthorization of the Education Sciences Reform Act.  Thanks.</P> <P>Frederick M. Hess:  Thank you, Russ.  Mike?</P> <P>Michael McPherson:  Thanks.  Well, I'm really happy about the existence of this conference and I think the large underlying question in general terms of how does research come to influence public policy is a terrific one to ask.  And I would love to see more comparative work as part of this.  Ken particularly touches on that in his paper but it will be really interesting to think about how this nexus looks in education policy and public health policy and welfare policy and tax policy, in abortion policy.  I have to say, compared to some of these other areas, things look pretty civil in the education arena.  And we need to be very careful not to idealize what goes on elsewhere in the policy world as we think about these things, including in medicine.</P> <P>The thing I want to emphasize is we really need to ask, in my view, the serious empirical question:  How does research influence policy?  But it is actually very hard and one can feel it even in these papers, I think, more so in Dan and Dom s than in Ken s.  And in Russ s comments, it is very hard not to slip away from that question of How does this actually work into the question of How can we make it work the way we want it to, which is, of course, a great question.  But most of the time we social scientists would say, Before you fix something why do you not figure out how it works?   </P> <P>Now, why is it so hard to keep our focus on the actual question of how does this world that we live in work in regard to how research influences policy?  There are some people who have made careers working on this problem.  Robert King Merton, Evelyn Bloom [phonetic] and David Cohen wrote a great book about it; Carol Wise has done work on it.  It is not a real active field these days.  </P> <P>And I think there are two reasons that it does not get studied more.  One is we are really asking extremely complex causal questions; influence is a causal idea.  And we know as social scientists how hard it is to address causal questions, and we know that this is a particularly complex causal situation.  People hold extremely complex sets of knowledge and belief structures.  And we really should ask ourselves, among all the things that influence knowledge and belief, where does systematic social science research fit in?  How does it change things?  Those knowledge structures get embedded in institutional structures, which are highly resistant to change and, therefore, they are going to reflect past research to a very large extent.  How do we expect those institutional structures to respond to change?</P> <P>So, we shy away from the difficulty of the problem, I think, and therefore talk about it is easier to talk about how to make things different than to talk about how they are.  The other thing, I think, is that we social scientists come at these questions with very strong normative preconceptions.  We believe that we should do great work answering causal questions and that these idiots should listen to us.  And the question is not so much what do they actually do as why will they not listen to us?  And it is very hard to stick with How does it actually work? </P> <P>So, I think it is great that we are spending a day bringing analytics and evidence to bear on these questions of how does research influence policy.  But recognizing, I think, these two papers and, really, all the work of the day are full of insights; I have learned a lot from them.  I want to push on two things that I have heard come up repeatedly.  </P> <P>I have not been able to be here all day, but when I have been I have heard these points come up.  One is this point, which, I think, is present in Dan s paper, of a certain tendency to conflate how influence actually works with how we would like it to work.  So the implicit model, I think, of influence that is present in Dan s and Dom s vision and in a lot of the talk today is that we want the world to be such that policymakers frame clear alternatives and seek evidence about which of those alternatives will produce the results that are desired.  And we think that the way to get those results is to do expensive random control trials, which will give definitive causal conclusions.  But if you look at the work of people like Jim March, Carol Weiss and others, they suggest a very different perspective that the work of social scientists often has its influence much more indirectly than that; not by answering the question the policymaker asked but by changing the questions.  By making people frame the problem differently.  </P> <P>Jim Rosenbaum, a wonderful sociologist of education, makes this point regarding a sociologist study of tracking in high schools, work that started in the late 50s and the early 60s.  And before that work was done, tracking was seen essentially as an engineering question.  Do people learn more if they are carved up in relatively equal skill groups?  And the sociological work showed that these structures become important structures of hierarchy and dominance and self-understanding in the high schools.  People who were put in the general track come to think of themselves differently.  That completely reframed the question, and it was no longer the case that the only question was, What produces the most book learning, but What do we do in [sounds like] the folks when we organize classes in this way?   That was the big change that the research produced.  </P> <P>So, I think we really need to spend more time thinking about how do people s understandings get influenced by what they read, and it is not as straightforward, I think, as a simple, They have a question; we will go get a causal answer, and therefore they will change what they do. </P> <P>The other thing I want to put in a word for is academic freedom; I'm sure we are all in favor of academic freedom.  But it makes me a little nervous to hear how skeptical we are about tenure and how important we think it is that people should orient their work to the public policy problems of the day as they are defined by major social institutions.  Clearly, there are big problems with the academic reward structure that we have.  It is not well-aligned with producing research that is of major social relevance.  All academic disciplines are subject to the kind of self-referencing and obsession with their internal structures that easily drive us crazy.  But I do worry about and too much interest in streamlining that process so that there is kind of a mainline connection between what we perceive social needs are right now and what the academy works on.</P> <P>Ever since the Middle Ages, academic work has always served two purposes from a social point of view.  One has been to feed the existing structures, whether it was producing the priests in the 15th and 16th centuries or whether it is producing the lawyers that we need, the public policy analysis we need or the research that we need in order to feed the existing structures.  But, always, parallel with that, the academy has served a critical role of questioning the prevailing values, of being that safe place where you can ask the question which is not being asked elsewhere.  </P> <P>And I think that both of these are important functions and, really, I mean, tenure serves this superficial political purpose of keeping people from getting fired for saying the wrong thing, which is certainly important.  But it serves the much deeper purpose of increasing the power of the faculty relative to the power of the administration.  And as a former college president, I can tell you it works.  </P> <P>There were days when I would go home and just say, I cannot believe those people are still going to have their jobs tomorrow.   But they are.  That effect is to say that what academicians study, what they judge to be worth caring about, is determined by their own developed academic values.  And we need to have people in the academy who are saying, Well, who said that better test scores are what we really need?  And why do we think that is the fundamental mark of what makes for a good education?   Should not somebody be asking the serious questions about are there other scales of value that we should be pursuing?  </P> <P>Now, that does not mean that there is not a lot of self-indulgent nonsense being done in the name of those kinds of critical concerns.  But I do not want to casually slide away from the independence of academic life, including the willingness which I hope will always preserve [indiscernible], to fund a certain number of things which do not have any obvious relevance to the next week s problems.  Thank you.</P> <P>Frederick M. Hess:  Thank you, Mike.  Kathy?</P> <P>Kathleen McCartney:  I want to thank Dan and Ken for their papers, which have led me do a little bit of thinking.  I'm going to organize my comments around two topics, really.  The first I'm going to call the use of data in a democracy, and the second will relate more to incentives, especially incentives within the academy.</P> <P>So, a number of people this morning have talked about the fact that social scientists are humbled, or should be, by the extent to which data are ignored in the political process.  And in a democracy, data exists alongside compelling testimonies from common ordinary citizens.  I think it was Daniel Patrick Moynihan who was the first person to say that values trump data.  So, what should the role of data in a democracy be?  </P> <P>Ken, when I was reading your paper, you asked under what conditions can researchers exercise greater influence in the policy process.  And Mike, I started thinking along the same lines as you, which is how should we, or should we?  Ken, you also quote Mary Jo Bane, who has influenced my thinking on this.  She said that researchers need to shift our perception from seeing ourselves mostly as expert problem solvers to seeing ourselves mostly as participants in democratic deliberations.</P> <P>So, in other words, decision-making in a democracy should reflect the political will of the people.  And I think William Howell was saying a little bit of this in the previous session.  I agree; I do not think our goals should be to influence policy by privileging our data.  Instead, I think, our goal should be to work to ensure that our data are part of the discussion versus, let's say, lying fallow in technical journals.  And I want to come back to that when I talk about incentives.  So, I think there are two obstacles that prevent researchers probably more, but I'm going talk about two from influencing policy.  And the first one, again, has been talked a lot about this morning and that is that the data are imperfect.  </P> <P>And by that, I think a number of us have meant that it is rare to have definitive data on any scientific topic.  But I want to make the point and, I think, again, Mike, you were making this - that this is not just the case in education science.  So, it is not just that there is no definitive data on class size.  I mean, in the last year there have been debates that I have read about as an educated person who reads the media on whether or not mammography is a successful screening tool for breast cancer.  I remember when I read that I thought it was some large meta-analysis and I thought, How can there be two sides to this question?   But there are.  Or how about two sides to whether or not coffee or wine is good for us?  I mean, every other year we are reading something different.</P> <P>Frederick M. Hess:  Wine is good for us.</P> <P>Kathleen McCartney:  Wine is good.  When there is clarity, things happen.  It took years and years and years for scientists to agree that smoking caused cancer.  And, again, back to the problem of establishing causality in a correlational world and I think that when there is clarity surrounding educational issues that I'm convinced that practitioners and policymakers will take notice.</P> <P>The second obstacle does have to do with Dan s point about the academy providing few incentives to market research.  I used to teach a course on child development and public policy and I always used Gina Kolada s piece on DARE.  Some of you are nodding so maybe you have used it in your classes, too.   But DARE does not work and, at least, when she was writing about it, 75 percent of schools used DARE to prevent drug abuse in schools.  In fact, in some schools, like upper-middle-class schools, it actually promotes the use of drugs because these kids get to see pictures of what they look like.</P> <P>Now, there are alternative more effective programs, and one was published.  It was supported by [indiscernible]; it was published in the American Journal of Public Health and it never found its way out of the journal articles.  </P> <P>So, Gina Kolada did some investigative reporting.  She talked to people at [indiscernible] and she talked to the researcher, and we know the answer.  We do not have to because we are social scientists.  There was nobody at [indiscernible] whose job it was really to get the word out and the researcher was on to the grant-funded project.  So, I think that is a problem, and again it is something I'm going to return to when I talk about what people in my position can do with respect to incentives.  </P> <P>But before I get there, I guess, as a social scientist, I do want to say one thing about how I think we all need to work in our role for our data to add value to any kind of political discussion.  And I think we have to work to make sure there are fair rules about how the data are used and not used, or even Dan, you were bringing up how the data are produced.  I mean, obviously, we need to produce some with integrity by not running scores of regression equations.  </P> <P>But in the political process of our data being used, I have noticed the rules changing.  It is not just that you need statistical significance now; you need an effect size that is big.  And you need an effect size that is big, depending on whether or not you are a Democrat or a Republican and depending on what side of the issue you fall on.  That is clearly a problem.  We need to decide ways of conveying what the practical importance of research findings are.  So, that is one rule of the game.  </P> <P>I think a second rule is we need to synthesize all research findings and not cherry-pick.  That is a real problem when you see some of these synthetic papers synthetic journal outlets that we are talking about.</P> <P>Frederick M. Hess:  Synthesizer.</P> <P>Kathleen McCartney:  Yes.  But I'm using synthetic as a double entendre.  I mean, all of the research findings need to be entered into any discussion, or what is the point?  I mean, the data are adding no value to rhetoric if you are allowed to cherry-pick.  I think the best way to make sure that the rules are fair is to encourage peer and public critique of the data.  And that means that the data need to be out there for a while before policy makers weigh in.</P> <P>One such example for me was when the Mathematica evaluation of the 21st century learning centers these were the after-school programs.  I mean, the day that those findings were released there were some Republicans on the Hill saying, Well, we have to cancel all money for after-school programs.   If you actually looked at the evaluation, there were serious problems with the implementation of the program.  Most of the kids did not go because they did not have transportation to the program.  I mean, things like this.  I mean, this is why peer and public critique are so important.</P> <P>How much time do I have, anyway?  Okay, I have a little bit of time.  Then I am going to talk a little bit about what we can do in the academy to make our data more useful.  Dan talked a lot about enhancing research rigor.  I'm a quantitative researcher, too, but I worry a little bit about the methods wars.  I do think this is a field where we need more experimental research, but not to the exclusion of other methods.  I wanted to say to all of you that I was a faculty member in two psychology departments before joining the School of Education in 2000.  So, I'm a little bit of an outsider and I have never seen a learning community as self-critical of its data as the education science community.  And I think that is a question that is interesting in and of itself.  </P> <P>But I think the field does need more experimental research.  I think it does need more attention to selection bias and that is something that has talked a lot in this community.  But I think there is a third thing that is needed, and that is large-scale studies that are built by collaborative teams; large-scale because usually you can answer better questions with large-scale studies.  </P> <P>Physics, I think, taught us this; especially, elementary particle physics has made enormous progress for two reasons.  One is funding, and I'm happy to hear you talking about this.  And I think there is a reason we do not know as much or, I would argue, we do not know as much about education as we do about physics but I think one of the reasons has to do with how we have privileged physics as an area of inquiry over education.  </P> <P>And I think we are beginning to see larger-scale studies in the social scientist.  I'm part of one.  I'm part of this NICHD study of early child care and youth development.  In the mid- 1980s, there were big debates about whether or not day-care disrupted the mother-child bond, whether or not day-care caused aggression, things like this.  And NICHD put out an RFA.  Ten people were funded and when they brought us to Washington, they said, We want you to work collaboratively rather than doing 10 individual research studies.   It was quite an experiment because we represented the gamut with respect to scientific orientation.  But I think that study has yielded great benefits.  </P> <P>I'll just give you one example.  The question of whether or not day-care disrupts the mother-child bond is no longer asked within the academy because once we did a study with 11,050 children where videotapes were coded in a double blind way and analyzed.  There was just no question.  And so there were people not on our team but within the field who said, We are done with this one.   And I think we could be done with some questions in education science if we did the same thing.  I have been thinking a lot about the role of education schools with respect to providing incentives so that the work we do will be, let's just say, more relevant to the policymakers.  And I think there are some mechanisms within the academy that can help.  I wanted to talk about just a couple of them. </P> <P> At my school, one of my colleagues, Tom Kane has been building a project for policy innovation in education and he is doing exactly what several people here have suggested.  He is bringing together people from districts with researchers and with funding agencies and they are getting in a room and talking about what problems need to be solved and then thinking about ways to solve them.  Jack Shonkoff is another person in my school who is working to develop a center for the developing child, the goal of which is to bring he talks about bridging the gap between what we know and what we do; and by that he means bringing the scientific research-based practitioners and policymakers in a non-partisan way.</P> <P>So, the last thing I wanted to talk about that I think schools of education can do is to reject Arts and Sciences models where success equals research productivity and to accept our role as professional schools where success should equal impact on policy and practice.  It is not as if and I want to talk about sort of the collective of a faculty.  I do not think you should expect each and every faculty member to do the same thing.  But I think it should be the case that a criterion for tenure could be impact on practice and policy so that it would not just be publishing in the top flight of economics journals that would be enough to get you tenure, being willing to come to Washington to give testimony before Congress or to go to the National Council of State Legislatures should be just as important.  Thank you.</P> <P>Frederick M. Hess:  Thank you, Kathy.  Dan?  And let me ask it struck me in the points from the paper that you emphasized just now, it struck me that most of them are actually not so much the incentives operating upon practitioners or upon researchers as they were some of the conditions on which it operates such as the standardization of assessments and outcomes.</P> <P>I think, Mike, when we talked about change in the incentives on the research in the ways we have all day encouraging them to be more relevant or choose your descriptor.  Mike and, I think, Jeff earlier this morning flagged some of the concerns about rubbing away the elements of the research institutions which protect the integrity of the research process.  I mean, do you want to talk a bit more about how we think about incentives and how do we, in changing incentives and researchers that might address some of the concerns today, how do we do that without undercutting the integrity of the research process?</P> <P>Dan Goldhaber:  I'm not sure which concerns you are talking about.</P> <P>Frederick M. Hess:  So, the notions that researchers are not have, historically, not been concerned enough with timeliness.  And that when they do get concerned with timeliness, we then get sloppiness and inattention that, left to their own devices, are not relevant enough and that they do not pay enough attention to, in Mike s phrase, socially defined problems.   But that if we encourage them to do that, that we risk this problem where they become extensions of contemporary debates and loose the ability to raise new questions or weigh in from outside.  So, say, take those two as two of the concerns.  If you just want to talk a bit more about some issues you raised in the paper and just help us think through that a bit more.</P> <P>Dan Goldhaber:  You know what?  Your characterization that I talked about the conditions, I actually think that changing some of these conditions do change the incentives.  So, when I was listening to the discussion about academia, I agree with everything that you said, Kathleen.  I'm just not sure in concrete terms what you do incentive-wise to change some of those things.  So, look, I think we would be far better off if we had more Tom Kanes, but Tom is the exception in the world.  </P> <P>So, I'm not sure how we get more Tom Kanes.  And, really, to be honest, I think that our conclusion was a bit of a there was not a lot there because it was hard to think about how you change universities.  So, the idea of a flagship journal if people took a flagship journal seriously and it really did factor into tenure, then it does not fundamentally change the university structure.  It just changes the incentives for the individuals in a school of education, for instance.  </P> <P>I also think that it changes the incentives for the reporter who is getting a lot of information, is on a timeline.  Actually, I have got to say the anecdote about IES was a little disturbing.  That was maybe came up before you arrived, but that you had offered to, sort of, vet the research for reporters and nobody took you up on it.  But I think that if reporters who are acting on deadlines and have their own set of institutional incentives if they knew that work published in the equivalent of JAMA was really topnotch important work, then I think that it changes their incentive, too, and so they sort of know who to go to.  Maybe they do not take two sides of a debate because they know that if they cannot sort it out themselves, that there is some credibility because work is published in this particular place and it is not debates about oh, I do not know whether a master s degree matters in education.  The weight of the evidence is not equal on both sides.  So, I think some of what we discussed about the conditions really do speak to the incentives.  I do not know if I answered your question.</P> <P>Frederick M. Hess:  No, no, no.  So, actually, I would like to take this a bit further.  Maybe, Russ can get into this.  When you say this is not enough time, there are just not many Tom Kanes, this is partly and what your interdisciplinary efforts to see the field why is it that there are not many?  I mean, it seems like there has been a lot of agreement that we would like to see more people with those skills working with districts that there are funders who would like to support them.  So, why is it that what he is doing is so unusual?</P> <P>Dan Goldhaber:  [Inaudible] why there are not more Tom Canes?  So, hold another conference, right?  But one thing that has happened </P> <P>Frederick M. Hess:  Kathy, make sure Tom knows that this came up.</P> <P>Dan Goldhaber:  That is next year s conference.  One thing that has really changed in the last 10 years is data.  So, there has been a revolution in the amount of data that is available and allows you to address issues in a more causally convincing way.  And so I think that that has drawn researchers into the area of education research in a way that they were not previously interested because they could not answer questions in, again, this causally convincing way.  So, I would say data is one of the major reasons why we have more people who are doing this kind of research today.</P> <P>Grover J. (Russ) Whitehurst:  I would add to that money and a fair and predictable way for getting it is part of the process.  So, you have economists like Dan who are making a decision as to whether it is the education sector or the health sector or some other sector that is going to be the grist for their mill.  Data is certainly important and so you go where there are data.  But you also go where you can compete for funds to analyze those data.  </P> <P>So, it has to be money there and if you have to think if I apply for funding to a particular agency and I'm good at what I do, there is a good chance that I'll be funded for it.  And that is why though it is not [indiscernible] nuts and bolts stuff having peer review system that is thought of as equitable and is fair as any human judgment process can be and having some money behind that that goes consistently to people who produce good work pulls folks in.  </P> <P>We have seen that in the cognitive sciences where we established a program on cognition and student learning that, I think, has attracted the strongest cognitive scientists in the country and has simultaneously moved them from doing all their work on freshman majors in psychology who are forced to participate in studies in order to get course credit to actually going out into schools and doing work that is increasingly relevant to practitioners.</P> <P>Kenneth Wong:  Yes, I was going to your last comment is very important, that is, we look at university professors as serving at least three roles.  One is researcher; the other is teacher; the third is doing some work for services that would impact the community.  And so the incentives can actually get at the other way; instead of giving more money to the researchers, you can actually create fellowship for inner-city youth to come to universities to study with good researchers so that you can actually train them and for the next generation, go back to the inner-city community and then make a difference.  There are different ways to think about incentives as kind of not so much directly involved with giving money to the principal investigator doing experimental design work, but also there are different ways to think about those combination of roles.</P> <P>Frederick M. Hess:  Okay, let's open this up.  We will have Juliet and Shane have mikes.  Do we have questions?  Yes ma am.</P> <P>Ann Schneider [phonetic]:  My name is Ann Schneider and I'm independently doing research with a grant from the Department of Education on International Education.  In the course of this, I have learned a lot about dissemination, which was required as a condition of the grant.  And I have learned a lot about how to write things for different audiences.  And I'm just wondering whether there is any possibility that some of that training could be included in courses on methodology?</P> <P>Frederick M. Hess:  Good question.  Kathy?  I guess, Russ?  Anybody?</P> <P>Grover J. (Russ) Whitehurst:  I'll just say that good clear writing is at a premium.  It is certainly at a premium.  And in federal government, I think it is in a premium in general.  And to the extent that people can be trained to do it in graduate school, they should be.  We have a lot of difficulty getting folks we fund to express themselves in terms that are comprehensible to practitioners.  And I think a lot of that is just a lack of experience in doing it and having models for it.  So, yes, I think some training in communication would be appropriate for a methodology course.</P> <P>Frederick M. Hess:  Now, I mean, how much of this is kind of Ken s two cultures playing out that, particularly, if somebody spent their early part of their academic career learning to write carefully and appropriately for a scholarly audience, the one becomes very used to making sure that everything is double-bolted protect yourself through the review process.  Obviously, that kind of writing does not work very well when we are talking about trying to write for your typical parent who might be reading this thing in a magazine or a newspaper.  And, in fact, by the time you have tried to reduce it to its essence in a way that somebody might actually read it in a magazine or on an op-ed page, it is likely that your colleagues will be uncomfortable with the way you have rendered your findings.  I mean, it seems to me all day long we have talked around this, but I do not know if there is any particularly good solutions to any of this.  Or is this just something we have to live with.  Mike?</P> <P>Michael McPherson:  No, I think that is actually a complex set of questions and I would put a lot of emphasis on I much prefer the word communication to dissemination.   I mean, I use the word dissemination, too, but it has such an implication of the center and the periphery and the heart of everything is the researcher, and it is just for the same reason I'm uncomfortable with translation as a metaphor.  These are very much two-way streets and I think we need to teach researchers to listen as well as to express.  </P> <P>But in my experience, I think this is clearly true in economics; I think it is true in other areas, too.  The people who communicate the best are the people with the deepest understanding.  And the people who I think of Steve Rowdenbush [phonetic], for example, who can even explain to me why some of these subtleties about experimental design makes sense, and that is quite an achievement.  And often, when people cannot communicate well without using the technical language of their discipline, it is fundamentally because they do not understand at a deep level what they are talking about.  </P> <P>So, there are bigger problems in educating people than simply teaching them communication skills.  Really, really understanding with the work you are doing helps a lot.  It is certainly not the whole solution, but it is important.</P> <P>Dan Goldhaber:  I think that is absolutely right.  But I also think that we should not confuse communicating well with communicating a simple message because so often the context matters; the detail of the study matters and it is not something is good or is bad.  And oftentimes that is the way the press wants to boil it down.  And researchers are pushed to say, is it good or is it bad?  And the answer often is it depends. It depends what you are comparing it to.  It is just a more complex issue and I do not know how we get it.  I mean, I think both parts are important.  I do not know how we get at the second part.</P> <P>Frederick M. Hess:  It is interesting.  It strikes me in light of the earlier conversations today is that researchers there is a point, right?  This is kind of Ingersoll s dilemma.  There is a point beyond which researchers are not comfortable going and, kind of, saying it is this or it is that.  They want to say, well and at some point somebody takes out the baton.  And so I think what Mike Petrilli and Paul Manna talked about, for instance, was on the teacher quality issue.  </P> <P>There are a couple of groups in that case, it was Fordham Foundation and Ed Trust and NCTAF but it could be others we could slide in there are a dozen groups like that.  And they obviously have an incentive to kind of frame the work in the way Kathy was alluding to and to cherry-pick [indiscernible] because they are not doing it out of some grand pursuit of truth; they are doing it because they have certain maps of the world in their head and they are going to read and interpret research in light of those maps.  </P> <P>So, in some sense the think-tank review project, for instance, [indiscernible] effort is trying to spotlight these guys in [indiscernible] research.  But much of what they are doing is they are in some sense acting as a bridge between sympathetic members on the Hill and in Washington to the research community out there.  And if they were not playing this bridging function, then I guess the question is there is a certain degree beyond which researchers are not comfortable going to the bridge by themselves.  </P> <P>So, can we think of better or less can we think of ways in which we do not have the concerns that Kathy raised in which we still are building those bridges?  I mean, I do not know.  Are there thoughts on that?</P> <P>Grover J. (Russ) Whitehurst:  Sure.  One way to do it is with dissemination vehicles that are have some of the subtleties smoothed out, I suppose, and are designed, particularly, for an audience of practitioners or policymakers.  We really do not have those in education to the degree they are present in health, for example.  The Institute will be issuing in the next couple of weeks the first of something called Practice Guides, which are our effort to provide a bridge between what is known about research and how it can be packaged and presented to policymakers so that they can make use of it.  </P> <P>So, a model I have used a lot and I do not know how I got to it.  You are Google-ing and you find stuff.  So, I was Google-ing around and I found the website of the Ontario Nurses Association.  And on that website, I find that they have a variety of practice guides that are intended to help nurses do their job.  And some of them were pretty terrible, but I liked the one on wound care on the extremities.   It gives eight things you should do if you have a patient with a wound.  And provides it is a classic and it provides an evidence grade for each of those recommendations.  And together, they are intended to be coherent, so these are the major things you would need to do.  Should you put on the wet bandage or a dry bandage?  How frequently should you change the bandage?  Antibiotic or not?  </P> <P>I think, perhaps, those kinds of products in education if they come from a source that is trusted to be playing it straight can have an influence on policymakers [indiscernible].</P> <P>Gregory McGinity:  Hi.  Gregory McGinity from the Broad Foundation.  On the previous panel, William talked a little bit about this, and you all have mentioned it both here and in your papers about the incentives for those going into academe before they get tenure.  Can you talk a little bit about the incentives after tenure?  Is it after sort of seven years of going through that or eight years going through that process, sort of looking at practical questions or working with practitioners or getting research placed in practitioner magazines, et cetera, is just been sort of beaten out of the candidates for PhDs or tell me just from your perspective sort of where those incentives are because we have not really talked about those.</P> <P>Kathleen McCartney:  I'll take this one.  Well, I just think there is huge variability in that.  After tenure, people are motivated, maybe, more intrinsically than they were pre-tenure.  Because there is just no question that if you are at a research university you are trying to publish in the right journals.  One of the things that I was suggesting, though, is that in a professional school like a school of education it might make more sense to in addition to research, teaching, and service, to add as a criterion, impact. </P> <P>Again, not that everybody would do it, but I think you want the incentives there so that people will do the kind of outreach work that really makes a difference.  And earlier, I mentioned this; we need to provide opportunities for people to want to go to the National Council of State Legislators.  </P> <P>Well, I did not pick that example randomly.  About six years ago, I think, I got a phone call from that organization:  Would I come and talk about our childcare findings?  And I thought, Well, of course.   These are the people who actually make childcare legislation; it is done at the state level.  But I did realize that I was going to have to change my PowerPoint slides so that they were accessible by a broad audience, not a scientifically trained audience.  And that takes some time; then I would have to fly to Denver and fly home.  And by the time you add it all up, it is about four days to do this 15-minute presentation, but you are getting the word to the right people.  </P> <P>Well, I get to Denver and I find out I'm the third person on this NICHD team that they asked because the first two said no.  The first two said no because they teach in psychology departments where there is absolutely no incentive to do this kind of work.  I found in transferring from a psychology department to a school of education that the cultural norms encouraged more of this.  But, still, people are protective of their research time if they like to do research; they are protective of their teaching time if they like to teach, and they are protective of their institution-building time if they enjoy institution building.  </P> <P>So, the long answer is that I think, post-tenure, a lot has to do with the individual and what the individual is motivated to do.  But I do think senior faculties at schools of education could make it explicitly acceptable to do the kind of outreach work that we are talking about.</P> <P>Frederick M. Hess:  Mike?</P> <P>Michael McPherson:  I actually thought about that same point in reading the papers because it must be the case that the vast majority of publications are from people who have tenure. And so you cannot except through some kind of socialization process you cannot explain that behavior by the tenure process as such.</P> <P>I think a big factor for people at leading institutions is the labor market.  There is a lot of mobility for the, say, some rough sense of the top 10 ed schools or the top 10 econ departments, top 10 et cetera.  And in the coin of the realm is prestigious publications.  And that traces back if you kind of it is what deans get rewarded for; it is how presidents pick deans.  I mean, somewhere deep in the roots of this is the NRC rankings of departments.  So, it is a pretty big system and it is not all bad, either.  I mean, it does we do have to remember that we do not just want lots of relevant stuff if it is not very good, and there is very intensive peer review assessment that is generated by both the tenure system and the vetting of senior candidates across departments.</P> <P>Grover J. (Russ) Whitehurst:  [Audio glitch] I think, focusing just on tenure.  And this is the fact that there are progressive set of cut points in academia.  There is tenure; there is promotion to professor; there is the chaired professor; there is the author from the other institution.  </P> <P>And not to be crass, but, money matters.  I found at least in the university where I spent a lot of time that when I was the department chair, the faculty bringing in the larger grants tended to get a break on various assignments and were privileged in a variety of ways.  And, certainly, I, as a department chair, was measured on the grant-getting ability of the faculty in my department.  So, it does come back a bit to money and, certainly, I will confess I thought about that at IES.  Can we make it more attractive for faculty to do particular types of work by allowing them to bring in significant dollars, thinking that after a while they will adhere to those who have those dollars the kind of prestige that currently adheres to more traditional scholarship?</P> <P>Frederick M. Hess:  Okay. Mike?</P> <P>Michael Feuer:  It is a short question.  I'm intrigued by what it came up here a couple of times and I do not know enough about it.  This thing, Russ, that you offered people to be able to call in and nobody came?  Could you say a little more about what your thoughts were in proposing that, how it worked, and what your inference is from the low response rate?</P> <P>Grover J. (Russ) Whitehurst:  There was a presentation I made at the Hechinger Institute at Columbia a few years ago, and it is an institute for the working press in education.  There were, I think, roughly 40 reporters in the audience who either were exclusively on the education beat or had that as a significant part of their responsibility.  </P> <P>And my presentation was about a theme that has been raised here earlier today: How the treatment of education research by the media is fundamentally different from the treatment of other types of research; it tends to be a human interest story instead of a reporting of the findings.  And part of our dialogue was about how difficult they found it to cover education research as they covered health research because there always seemed to be another point of view, and it always seemed to them to be advocacy of one sort or another.  </P> <P>So, if you think it is advocacy, you have a responsibility as a reporter to balance the point that has been presented in the paper with somebody else s point of view.  And I said, I have got a solution for you.   And, actually, I led them to this because I had the handout already and it is a fast response review system by the Institute of Educational Sciences.  We will give you a methodological read on the quality of the study within 48 hours after you ask us to do it.   </P> <P>And we set up an e-mail account and they each had a flier and I understand that Hechinger subsequently contacted those members of the education press who were not present to advertise the availability of this service; again, there was not a single taker.  Now, it has been so long that we have extinguished on looking; maybe the inbox is full.  </P> <P>So, there are other things.  I think there has to be another approach.  One of the problems here is, of course, the print media have sort of been decimated.  And so people papers that used to have full-time folks on the education desk are now covering topics that are all on deadlines.  They do not, I think, by and large, have time to deal with this as seriously as you would hope.  One of the portions of the contract, we are now competing for the re-award of the What Works Clearinghouse, has a mechanism where when a paper comes out, whether it is from a think tank or academia, that is generating significant public interest and I do not know quite how we would define that quite yet.  But when that happens, it will trigger a fast review by the Clearinghouse on methodological quality in order to try to contribute to the public debate by taking into account the quality of the study of the study, at least, on the dimensions that can be systematized, which have had to do with the degree to which the question asked has, in fact, been answered with the methods that have been deployed to answer it.</P> <P>Frederick M. Hess:  Mike, The Spencer Foundation has been thinking about these questions a bit.  Do you have a quick thought or two you want to add on this?  Questions of relationship between research and journalism?</P> <P>Michael McPherson:  Well, yes, we have thought a lot through a couple of years of board retreats and such about education journalism.  And I think my take and, I think, probably where we are headed as a foundation is that not so much to focus on the reporting of specific findings.  What I actually worry about more is the difficulty that both reporters and readers of, say, sort of elite news organizations - the New York Times, the Washington Post, the Atlantic Monthly is putting a particular finding in context.  Is this a big deal or is this a blip?  </P> <P>So, my mantra is where do we get the Linda Greenhouses of education journalism who can tell us which Supreme Court rulings actually matter?  So, we are thinking hard about how do you get some of the really capable journalists who exist these days think of Malcolm Gladwell as an example who really are good at putting ideas in some kind of context and seeing how they hang together.  How can you get them more interested in education as a subject because I think it is a subject that is right for that kind of really skillful exposition?  There is some of it out there; we would love to see more of it and we are trying to think about responsible ways to encourage it from a place that is not exactly completely rolling in dough.</P> <P>Frederick M. Hess:  Dan, Ken, Kathy, any final thoughts?</P> <P>Kenneth Wong:  One final thought is that I was pleased to hear that Mike offered, as president of the Spencer Foundation, that you would like to fund research in the area of institutional analysis.  I think that is very, very important and I hope that you will issue an RFP so that we can all take a look at that.</P> <P>Frederick M. Hess:  Always dangerous to make a statement.</P> <P>Dan Goldhaber:  The only thought is just to comment on something Russ said and that is that I wonder if the reporters did not respond immediately because there was not something public where they could say that there is no it is not that when they are reporting they could say, This had the stamp of approval of , and that the re-competition of What Works Clearinghouse sounds like it is going to be that vehicle and, maybe, that will get a better reception.</P> <P>Frederick M. Hess:  Any final word?</P> <P>Kathleen McCartney:  No.</P> <P>Frederick M. Hess:  Okay.  With that let me first, let me keep the panel up here for a second.  First, I would like to thank Juliet Squire for pulling this together, and Shane and Rosemary and Morgan for all that.  I also would like to thank the Spencer Foundation and all the generous funders who support the AEI ed program for making this program possible.  </P> <P>With that, again, keep your eye out; the book will be available early 08.  The papers will be available on-line through that.  In particular, in that context, the papers are on-line now.  Hard copies are out there.  You have e-mail contact for all of the authors in your folder.  We have a lot of expertise in this room.  As you have the chance to read the papers, if you have thoughts I would encourage you to please feel free to contact the authors and share thoughts and insights and challenge them as need be; it always makes for a stronger volume.  And feel free, obviously, to do the same with me.  Although, since I do not know as much as the authors it is always easier for me to respond.  I would also like to thank all the authors and discussants today, including the final panel that is up here.  </P> <P>So, thank you very much.  And with that, I would like to invite all of you to join us right out here.  We have got, I believe, some refreshments and wine set up.  So, please join us and feel free to relax.</P> <P>[End of file]</P> <P>[End of transcript]</P></body></html>