Rigor and transparency: Required for approval of drugs — needed for certification of providers

Quality is the new watchword in healthcare; it's what we seek - and increasingly, what we try to measure.  Medications, devices, care delivery, hospital services - all are now scrutinized as we seek to gauge their benefit, and justify their cost.

The idea of using metrics to evaluate quality make sense, but only if we can trust the metrics themselves.  Otherwise, we risk becoming party to an updated version of craniometry, systematized false-precision that focuses on easily-measurable parameters (such as head circumference) that may not represent meaningful proxies for the assessments we're really after (i.e. intelligence).  

"We're now better able to recognize the qualities and properties of good tests - and to identify where they're likely to fall short." -David Shaywitz, M.D.The good news is that the science of testing, of developing evaluation instruments, has improved over time.  We're now better able to recognize the qualities and properties of good tests - and to identify where they're likely to fall short.

We're also getting more comfortable with demanding robust evaluation instruments.  For example, the FDA's approach to patient-reported outcomes places exceptional (and appropriate) emphasis on the assessment tool chosen, and requires that it demonstrates the appropriate properties before relying on its results.

Unfortunately, one critically important area within our healthcare system that seems to have escaped such careful review is the way the competence of care providers is typically assessed and certified.

Whether you are an X-ray technician, a physical therapist, a registered nurse, or a transplant surgeon, you are required to pass through a gauntlet of costly certification exams.  These tests, already significant, are assuming an even greater importance as the healthcare system increasingly looks to them as proxies for quality.  Certification can be required for employment and for admission privileges, and frequently impacts the reimbursement rate for healthcare providers.

All this makes complete sense - provided the certification tests themselves are sound.

Unfortunately, the world of healthcare worker certification remains a bit like the wild west, as medical organizations and professional societies approach certification testing with profoundly different degrees of rigor - and generally little-to-no transparency.

An assessment test, fundamentally, should be judged by two criteria: reliability and validity. 

Reliability refers to the intrinsic quality of the measurement instrument itself: Does it yield consistent results, independent of who is scoring it?  Would the same person, if administered the test on different occasions, likely receive the same result?

Validity requires reliability, but represents a higher bar, and speaks to the relevance of the test.  Does the exam measure what you think it does?  Does a healthcare worker certification test accurately predict competence?

With the right tape measure and technique, for example, head circumference might represent an extremely reliable instrument for the measurement of intelligence, but obviously would not represent a valid tool.

High stakes, widely-administered standardized tests such as the SAT and MCAT are generally viewed as reliable instruments, although with limited validity; they are best at predicting first year performance (generally less accurately than high school or college grades, respectively), as well as performance on subsequent standardized exams.

As you move into tests that are given to more specialized groups of people, not only is it even harder to find any meaningful evidence for relevant validity, but even the reliability of the instruments is often (though not always) uncertain, or inadequately substantiated. 

As a result, professional societies administer - and pocket hefty fees from - tests that may not be reliable, and which are used as proxies for quality yet may not forecast better than chance, and may not be able to meaningfully distinguish between examinees.  The competent may be labeled as unqualified, while the dangerous may be accepted as competent.

Often, there is little incentive for those who administer the tests to put in the effort required to produce a reliable instrument.  To this point, most stakeholders have been content to take the word of professional societies regarding who is and isn't qualified, assuming these expert organization are best positioned to make this determination.

As our healthcare system has come under increased scrutiny, however, one of the great lessons is that we no longer need to accept the word of authority figures, and should insist on transparency, on verifying the data for ourselves. 

We now know enough about assessment instrument development that we can demand any professional society offering (and charging steep fees for) certification exams provide data demonstrating these exams are at least reliable instruments - and beyond that, correlate in some way with the outcome we seek.  It can no longer be acceptable for certification organizations to hide behind a fortress of assumed authority.

Guilds such as medical societies fundamentally can select their members any way they choose, and can select their own inclusion criteria, whether arbitrary or rigorously validated.  Consequently, responsibility for ensuring the certification process robustly correlates with professional quality and real-world performance likely rests outside the guilds, with those who must decide how much value to place upon guild certification.

At a minimum, no hospital or health system should use certification as a basis for employment for any healthcare worker unless there's clear evidence the certification itself is meaningful; payors (including the government) should similarly decline to provide a pay differential for certified vs non-certified healthcare workers without clear evidence the certification process is robust, and demonstrably correlated with quality. 

It's concerning to contemplate how much federal money may flow to healthcare workers based on potentially flawed certification processes, and how many patients may be routinely exposed to workers who are not appropriately qualified. 

Used properly, metrics - including, but in no way limited to certification testing - can significantly improve the quality of our health, provided the measures used are rigorously developed and transparently validated.

But numbers for their own sake, without an appropriate foundation of support, are worthless.  Actually, they're worse than that: they're harmful, as they obscure rather than illuminate, and make true quality even more difficult to discern.


Also Visit
AEIdeas Blog The American Magazine
About the Author


  • Dr. Shaywitz trained in internal medicine and endocrinology at MGH, and conducted his post-doctoral research in the Melton lab at Harvard. He gained experience in early clinical drug development in the Department of Experimental Medicine at Merck, then joined the Boston Consulting Group’s Healthcare and Corporate Development practices, where he focused on strategy and organizational design. He is currently Director of Strategic and Commercial Planning at Theravance, a publicly-held drug development company in South San Francisco. He recently wrote Tech Tonics: Can Passionate Entreprenuers Heal Healthcare With Technology? 

  • Email: davidshaywitz.aei@gmail.com

What's new on AEI

image Recovering from tax time blues
image 10 welfare reform lessons
image Let HHS nominee Sylvia Burwell explain Obamacare lie
image Why bold ideas backfire in politics
AEI on Facebook
Events Calendar
  • 14
  • 15
  • 16
  • 17
  • 18
Wednesday, April 16, 2014 | 10:00 a.m. – 11:00 a.m.
Calling treason by its name: A conversation with Liam Fox

Join us at AEI as the Right Honorable Liam Fox sits down with Marc Thiessen to discuss and debate whether America’s intelligence agencies have infringed on the personal privacy of US citizens.

Event Registration is Closed
Thursday, April 17, 2014 | 4:00 p.m. – 5:00 p.m.
The curmudgeon's guide to getting ahead

How can young people succeed in workplaces dominated by curmudgeons who are judging their every move? At this AEI book event, bestselling author and social scientist Charles Murray will offer indispensable advice for navigating the workplace, getting ahead, and living a fulfilling life.

No events scheduled this day.
No events scheduled this day.
No events scheduled this day.
No events scheduled this day.
No events scheduled this day.
No events scheduled this day.