It started as a simple story. Once again, the Trump administration had demonstrated its reputed disdain for facts and evidence. This time it had revoked the contract of one of the federal government’s top evidence clearinghouses — one that reviewed studies of mental health and drug treatment programs to determine their effectiveness.
The decision, which came quietly during the holidays, seemed to further prove that the administration cared nothing about facts, nothing about evidence, and little about evidence-based approaches to opioids, which it had elevated to a White House-level priority.
And then the story began to fall apart. An independent review of the clearinghouse had revealed substantial problems with its ratings, including significant potential conflicts of interest. Elinore McCance-Katz, the newly confirmed director of the Substance Abuse and Mental Health Services Administration, echoed those criticisms in a strident public statement.
But underneath the charges and counter-charges, there was a quieter, lurking story — one of widespread problems and alleged corruption in medical research. It is an important story, one that could be a harbinger of growing threats to the evidence-based movement as a whole.
SAMHSA’s evidence clearinghouse, the National Registry of Effective Prevention Programs (NREPP), was first created in 1999 in the wake of growing interest in evidence-based medicine. It was an early federal foray into evidence reviews, coming years before other federal clearinghouses like the What Works Clearinghouse at the Department of Education.
After its initial creation, SAMHSA’s clearinghouse was modified and revised several times, most recently in 2015, when its screening criteria were updated. In the aftermath of these changes, the number of programs reviewed and included in the clearinghouse grew rapidly.
This growth drew the attention of Dennis Gorman, a professor of epidemiology and biostatistics at Texas A&M, who examined the underlying studies that NREPP had used to make its decisions. In 2017, he published an article in the International Journal of Drug Policy that sharply criticized the clearinghouse for its poor quality standards.
According to Gorman, the large majority of the new programs approved by the clearinghouse were based on questionable studies, with most suffering significant conflicts of interest, including:
- Single Study Approvals: Of the 113 approved new programs, more than half (67) were approved on the basis of a single published article (51), non-peer-reviewed online report (4), or unpublished report (12). Fewer than half (46) were based on two or more published reports.
- Questionable Methodology: Many of the studies featured common and easily identified design flaws, including very small and non-representative samples, high rates of study attrition, and brief length of follow-up.
- Conflicts of Interest: Most of the approved programs (87) were based on studies or materials that included someone who was associated with the studied program as the author or co-author.
Today, with an ever-growing list of programs that claim to be evidence-based, clearinghouses are intended to be a stamp of approval, allowing users to sort the wheat from the chaff. But, according to Gorman, this was not happening:
As the number of programs grows, these [problems] are increasingly difficult to identify.
Worse still, the current NREPP review process essentially equates any such quality interventions with those that have been evaluated by the individual who developed and disseminates the program using a very small, self-selected sample, and in which the findings of the evaluation have appeared only in an internal report or an unpublished manuscript or a pay-to-publish online journal.
It even includes interventions that employ therapeutic practices such as thought field therapy and eye movement desensitization that are considered potentially harmful and supported only by pseudoscience.
Gorman suggested the following changes to the clearinghouse’s procedures, which SAMHSA may (or may not) consider in the aftermath of its decision to cancel the contract:
- Improving the transparency of its review process;
- Providing detailed declarations of financial conflicts of interest among program developers who review their own programs;
- Requiring truly independent replication studies;
- Assigning most significance to results from studies appearing in journals that adhere to rigorous publication standards, such as requiring preregistration of analysis plans and data and materials sharing; and
- Putting a mechanism in place (such as Registered Reports) that clearly distinguish exploratory research from hypothesis testing.
Patrick Lester is director of the Social Innovation Research Center.