Opened Data: Analysis of Requests for NHLBI Trial Data May Allay Some Concerns

Amid the hue and cry over the International Committee of Medical Journal Editor’s (ICMJE’s) plans to require data-sharing as a prerequisite for publication, one medical journal has elected to ask the question: just what do investigators look at when they access other researchers’ data?

According to an analysis of requests made to the National Heart, Lung, and Blood Institute’s (NHLBI’s) data repository, the vast majority of research using data from previously conducted clinical trials is targeting new questions and new secondary analyses.

“Requesting data for the a priori purpose of reanalysis or verification of original findings was rare,” according to Sean Coady, MS (Division of Cardiovascular Sciences, NHLBI, Bethesda, MD), and colleagues. Results of their analysis were published online March 29, 2017, ahead of print in the New England Journal of Medicine.

As previously reported by TCTMD, reactions to the ICMJE’s data-sharing proposal ran the gambit from fulsome praise to fury and disdain. Naysayers—many of them clinical trialists—were particularly concerned that nontrialists getting their hands on data would misunderstand or misuse it. Some also expressed fears that animosity between academic groups could lead to one group redoing analyses and challenging results. Indeed, an editorial published in NEJM by Editor-in-Chief Jeffrey Drazen, MD, and deputy editor Don Longo, MD, when the proposal was first floated touched off a firestorm when it used the term “research parasites” to describe some of the potential repercussions.

Asked why he and his colleagues undertook the review of NHLBI data requests, Coady said: “Dr. Drazen essentially asked us to, to help inform him of basically what happens once data is widely shared, largely without restrictions.”

Sharing the Wealth

Coady and colleagues reviewed data requests logged between January 2000 and May 2016; the NHLBI’s formal data repository, created to facilitate data-sharing, was launched in 2000. Coady, a statistician, has been managing the data repository since 2002.  

Over this period, 370 investigators requested data from one or more NHLBI clinical trials—with more than half of those requests occurring within the last 4.4 years. Almost three times out of four, those requests have been for post-hoc, secondary analyses of new questions, while 9% have been for analytic or statistical approaches to clinical trials and 7% have been requests for meta-analyses or pooled studies. Just 2% of requests have been with the aim of reanalyzing primary outcomes findings.

Asked whether “analytic or statistical approaches” constitute an effort to redo study analyses for the sake of verification or challenging results, Coady said no. These were typically efforts to assess whether the trials could have enrolled fewer patients or used different statistical approaches.

Any studies conducted, seemingly, as “another set of eyes” or to reproduce results “all essentially confirmed the original results,” he clarified.

Reasonable Expectations

Coady and colleagues took the additional step of reviewing any comments or editorials that accompanied any of these secondary publications. They concluded, based on these formal outside opinions typically commissioned by the publishing journal, that the studies had no major problems and received the same level of praise or criticism that also met the primary publications by the clinical trial investigators. As to whether these secondary analyses were achieving similar levels of high-impact-factor journal publication, Coady said he and his colleagues have not specifically looked at this but added that, in general, these secondary analyses tended to be published across the same range of medical journals.

Asked whether there are reassurances in this NHLBI analysis for those who have been critical of the Open Data movement, Coady acknowledged that he does understand the concerns trialists have about misuse.

“I can’t completely alleviate their fears and concerns, but our experience suggests that the secondary analyses that take place, at least with our data, tend to be reasonably well done, and publication metrics suggest that these papers are about as good as the papers we support as a whole.”

On the other side, for proponents of data-sharing who have advocated for swifter, broader sharing of clinical trial results, Coady had some caveats. While the data come into the NHLBI repository “quite clean” there is a cost to doing this well, including the time taken to document algorithms and endpoint definitions, as well as patient confidentiality requirements. The NHLBI, for example, requires the assignment of new patient IDs and the removal of any dates or other information that could unintentionally unmask participants.

More broadly, he continued, “As an institute, encouraging data-sharing and public availability of data has been a long-term commitment, but we also recognize that investigators, who planned the study, evolved the protocols, and went through all the [quality control] monitoring, . .  . have a reasonable expectation that they will get a chance to interrogate the data that they spent so much time and effort collecting.”

The NHLBI’s data repository is opened to outside requests after a “proprietary period” of about 2 years.