COVID-19 Blamed for Weaker Research Published by Top-Tier Journals in 2020

Since COVID-19 blazed its way across the globe, there has been “unprecedented scientific production” from the research community, with thousands of papers published in the first 4 months of 2020 alone, but their overall quality has been less impressive, one researcher says.

A new analysis by Giulio Stefanini, MD, PhD (Humanitas Clinical and Research Center IRCCS, Milan, Italy), and colleagues indicates that the quality of evidence published in the New England Journal of Medicine, Lancet, and JAMA was lower in these months as compared to the same period in 2019 and that the decline can be pinned on COVID-19.

Stefanini presented the analysis during a late-breaking science session at the virtual European Society of Cardiology Congress 2020. Speaking with TCTMD, he stressed the analysis is not intended as criticism of these journals—all were bombarded with papers when the virus struck—but rather a reminder for physicians to be vigilant when they interpret new research. During a pandemic, the stakes are high and both the quantity and quality of published studies need to be factored into clinical decision-making.

The issue is that this is an unknown disease and there was an urgent need to have some evidence, because some evidence is better than no evidence. Giulio Stefanini

“The lines of evidence that have become available need to be interpreted with caution because we all know that studies published in major medical journals, particularly during an emergency such as COVID-19, are translated into clinical practice within a couple of days,” said Stefanini.    

In the early months of the pandemic, there was a lot of confusion about COVID-19 and several major journals published studies with conflicting evidence, he noted. For example, there were reports questioning whether patients with hypertension might be at increased risk of infection due to their use of renin-angiotensin-aldosterone system (RAAS) antagonists, while other studies found no evidence of harm. One high-profile study published in the New England Journal of Medicine on the safety of ARBs and ACE inhibitors was later retracted and is largely believed to have been fabricated. Other conflicting studies with drugs such as hydroxychloroquine and remdesivir (Gilead) also were published in major journals.

More Observational Studies and Case Reports

With this mind, the researchers wanted to evaluate the quality of original research published between January and April 2020. They focused on the three highest-ranked medical journals and assessed quality using four parameters: randomized design, defined primary hypothesis, defined primary endpoint, and GRADE quality of evidence score. Overall, the group screened 3,370 articles identified on PubMed, including 1,805 published in the first 4 months of 2020 and 1,565 during the same time period in 2019. Of these, there were 339 and 297 original research articles published in these journals in 2020 and 2019, respectively.

Fewer studies were randomized in 2020 than in 2019 (29.2% vs 41.4%; OR 0.58; 95% CI 0.41-0.82). In contrast, 25.6% of original research articles published this year were observational studies, compared with 18.2% published in 2019 (OR 1.55; 95% CI 1.04-2.32). Similarly, 41.9% of the articles in 2020 reporting original research were case reports, whereas the proportion was 32.6% in 2019 (OR 1.48; 95% CI 1.06-2.08).

Additionally, the study hypothesis and primary endpoint were less frequently defined in studies published in 2020 versus 2019. The GRADE score also showed that 2020 saw the publication of weaker studies. Using GRADE, just 13.7% of 2020 studies were considered “high quality,” compared with 27.6% of studies published in 2019 (OR 0.41; 95% 0.27-0.63). There was an offsetting increase in the publication of studies with “very low quality” in 2020.

In a sensitivity analysis that excluded COVID-19 research, the researchers observed no difference between the quality of original research published in 2020 compared with the year prior. When they restricted their analysis to cardiovascular research alone, they also observed no difference quality.

Unprecedented Volume

To TCTMD, Stefanini reiterated their new data are not intended to bash journals for publishing lower-quality COVID-19 research, noting that the editors were receiving and reviewing an unprecedented volume of papers during the pandemic. While some journals might receive 30 new submissions a week, one journal editor told him they’d received as many as 200 new articles in a single day. Research published earlier this week in Mayo Clinic Proceedings found that review times have been an order of magnitude shorter during the COVID-19 pandemic.

Stefanini said he believes journal editors attempted to provide a service to their readers, even though the quality of those studies might have been lower than usual, because of the public health emergency caused by a new disease. 

“The issue is that this is an unknown disease and there was an urgent need to have some evidence, because some evidence is better than no evidence whatsoever,” said Stefanini. While the data published in those initial months is important, he added, “a study needs to be evaluated in light of its limitations.”

In terms of their own limitations, Stefanini said they focused only on three medical journals and the results might not be applicable to other scientific publications. The GRADE system to assess quality does have “articulated, comprehensive, and systematic methodology,” though it is also not exempt from bias. Finally, the group only analyzed data published between January and April, when knowledge about COVID-19 was limited. Whether the quality of published studies has improved is not known and would need to be studied, he said.