Low Numbers and ‘Anecdata’: Shoddy Research Rode COVID-19’s First Wave

The most rigorous effort to date to gauge the quality of research related to COVID-19 published in the pandemic’s first wave has confirmed the grim findings of earlier reports: the vast majority of papers lacked original data, adequate numbers, or weren’t reliable due to other, significant study flaws.

Even among clinical, peer-reviewed original articles, which included cross-sectional, simulation-based, and cohort studies, a critical assessment using validated tools revealed that most had methodological flaws that put them at a medium to high risk of bias. For example, among the 306 cross-sectional studies published in the first few months, 83% were deemed to be at high risk of bias because their sample size was too small to make definitive conclusions or because they had a low completion rate.  

The new review, which was led Marc Raynaud, PhD (Paris Translational Research Epidemiology and Biostatistics Department, France), suggests that traditional publishing standards for scientific research had lapsed in light of the COVID-19 pandemic.

“The study is not to blame anyone,” senior investigator Alexandre Loupy, MD, PhD (Paris Translational Research Epidemiology and Biostatistics Department), told TCTMD. “Within the scientific and medical community, as the virus started to spread, everyone wanted to do something. There was a race to publish, and the journals were faced with a flood of papers. It was an epidemic of scientific papers within a global pandemic. It was unprecedented.”

Loupy stressed that sharing data in the midst of a pandemic is truly important, and said he understood the desire to get information out there as fast as possible.

“However, to be useful for other physicians and researchers, any study should carry a minimum [degree of] quality,” he said. “We unfortunately doubt that studies with few patients, suboptimal design, or scarce data had any value to the clinical care. It is obviously very difficult to strike the right balance, but I think the scientific standards should be maintained to avoid contradictory results as much as possible and to prevent the peer-review process from losing quality.”

Cardiology Journal Approaches

In light of the new study, as well as earlier reports challenging the quality of published COVID-19 research, TCTMD reached out to a sample of cardiology journals to hear how decisions were made.

At Circulation, Executive Editor James de Lemos, MD (UT Southwestern Medical Center, Dallas, TX), said that during the early days of the pandemic, particularly when the virus was ravaging China and Italy, they felt they had an obligation to make the limited information available so that cardiologists would understand what they would be facing. At the same time, they had a responsibility to maintain scientific standards, and to protect the reputation of the journal.  

It was an epidemic of scientific papers within a global pandemic. It was unprecedented. Alexandre Loupy

Performing any type of study during a pandemic is heroic, de Lemos noted, which is what researchers were doing in China, Europe, and elsewhere. But the editors were aware these early studies were likely to be relatively weak from a methodological standpoint. So with all of that in mind, Circulation editors decided to proceed slowly with publishing COVID-19-related research, opting not to publish full-length research articles in the early phase and instead focusing on expert opinion and perspective articles from physicians. They also elected to publish COVID-19-related case series, which they would rarely ever publish in Circulation, because of their importance to physicians.

“We’re generally very skeptical of the general area of observational comparativeness research, meaning the ability to compare treatment outcomes without randomization,” said de Lemos. “The journal has a very high bar for that and we tried to maintain that even during COVID-19. We weren’t going to publish observational data suggesting that treatment A or treatment B might be better for COVID-19 or its cardiovascular complications. In general, that kind of research has many problems and oftentimes isn’t verified by randomized, controlled trials. We weren’t going to put something out there that we didn’t believe was true.”

Similarly, Brahmajee Nallamothu, MD (University of Michigan, Ann Arbor), editor-in-chief of Circulation: Cardiovascular Quality and Outcomes, said their guiding principle with respect to COVID-19-related submissions was to do no harm, even as the floodgates opened.

“Almost immediately, we began to see a spike in paper submissions," said Nallamothu. “It was incredible the number of submissions we received.”

Along with Deputy Editor P. Michael Ho, MD (University of Colorado School of Medicine, Aurora), Nallamothu tried to focus on publishing articles that would have a positive impact on clinical care. To select papers considered trustworthy, Nallamothu and Ho set up a system to handle the increased volume, one that engaged associate editors and editorial board members to help in the review process and to solicit experts that could provide meaningful insight about the impact of COVID-19 on the heart and vascular system. 

David Newby, MD, PhD (University of Edinburgh, Scotland), the deputy editor of Heart, also witnessed the ever-expanding research related to COVID-19 in 2020, as well as the obvious rush to publication. A lot of data that came out early in the pandemic did so because physicians and researchers around the world needed to share what they knew, he said, which could only help improve the understanding of the disease. Inevitably, though, there was a downside.

“There is a pressure to get things out quickly because everyone is working on it,” said Newby. “I think some of what was published early was poor quality. Certainly, some of the papers we’ve seen are just full of bias, sometimes acknowledged and sometimes unrecognized, but because the science is in such a hurry, attention to detail [is lacking]. People want to get the paper out and being more considered takes more time.” 

Y.S. Chandrashekhar, MD (University of Minnesota, Minneapolis), the editor-in-chief of JACC: Cardiovascular Imaging, said that the number of submissions to the journal was roughly 40% higher in 2020 than in previous years, an increase that included COVID-19-related research as well as studies unrelated to the virus. Early on, though, he proposed the journal maintain its strict scientific standards for publication, a decision that was supported by the editorial board. 

“I had a very negative view of the papers right from the beginning,” he told TCTMD. “When these things happen and people want to do the right thing and get data out there, very often it’s half-cooked and it’s tough for us to make good use of the data. . . . We decided we weren’t going to take COVID papers unless they were absolutely outstanding.”

Relaxing scientific standards, he said, makes sense if it helps society, but “if it mucks up the science and people do things they would not normally do, then we’re causing harm.”

Buyer’s Remorse

And in fact, some high-profile medical journals made some high-profile flubs, which might be expected considering the volume of studies they were reviewing.

For example, the New England Journal of Medicine retracted a study led by Mandeep Mehra, MD (Brigham and Women’s Hospital, Boston, MA), and colleagues showing that the use of ACE inhibitors or ARBs was not associated with an increased risk of death in patients with COVID-19. That 9,000-patient observational study of electronic health records was retracted following questions about the legitimacy of its data source. Mehra, along with Frank Ruschitzka, MD (University Hospital Zurich, Switzerland), and Amit Patel, MD (University of Utah, Salt Lake City), also retracted a paper in the Lancet, that one a global analysis of 96,000 patients with COVID-19 showing that chloroquine and hydroxychloroquine increased the risks of mortality and cardiac arrhythmias when used in the treatment of hospitalized patients with COVID-19. Again, there were concerns about the legitimacy of the data.

When these things happen and people want to do the right thing and get data out there, very often it’s half-cooked and it’s tough for us to make good use of the data. Y.S. Chandrashekhar

In both cases, the company that owned the database used in the analyses, Surgisphere Corporation, would not provide independent researchers with the raw data necessary to validate the findings, Mehra said, so they were forced to retract the papers following questions raised by other academics.   

Newby pointed to the example of several high-profile case reports published in the early months of the pandemic highlighting a heightened risk of myocardial damage in patients with COVID-19. Early MRI data, including one study that was criticized on social media, suggested lingering heart damage in nearly 80% of patients who’d recovered from COVID-19. Based on these earliest case reports and studies, cardiac abnormalities in COVID-19 were largely overestimated, said Newby. Concerned about the media attention these reports garnered, an international group of physicians signed an open letter urging cardiology societies to discourage cardiac magnetic resonance (CMR) screening for COVID-19-related heart abnormalities in asymptomatic patients. Better studies, including a global echocardiographic survey led by Marc Dweck, MD, PhD (University of Edinburgh), on which Newby was a co-author, subsequently showed that new MI, myocarditis, and Takotsubo cardiomyopathy were observed in only a small minority of patients.

de Lemos said he believes that every journal, including Circulation, made compromises to editorial processes during COVID-19 in terms of the quality of original research published. Given the circumstances, that bar had to be lowered because providing some information to practicing physicians was absolutely necessary, and every journal editor, he suspects, has some “buyer’s remorse” for studies they wish hadn’t seen the light of day. At Circulation, the editors tried to adopt a very intentional approach to avoid some of the pitfalls that could happen if the bar for publication dropped too low.

“We knew the worst thing we could do was publish wrong clinical data,” said de Lemos. “There’s nothing worse than publishing something unlikely to be true. At the same time, it’s a bit of a wobbly bar because some information is needed.”

That urgent need for information is driving another problem that’s become emblematic of the COVID-19 era, Newby noted. “We get the same sorts of things sent to us again and again and again. I think its people’s desire to get the knowledge out there, but it can get a little carried away. I think we need to be a little bit more thoughtful.”

COVID-19 papers, he added, are widely cited, but much of it is “repetition,” and many of these studies have “been badly done in a hurry.”

Methods in the Madness

Nallamothu acknowledged the flood of “anecdata” published in the first wave but defended its usefulness in light of the evidence gap, adding that these decisions were fluid and changed over time. For example, Nallamothu and Ho published a study on the impact of the COVID-19 outbreak on STEMI care in Hong Kong. That paper, with just seven patients who underwent primary PCI, would have been quickly rejected under normal circumstances; it was the first, however, to highlight the effects on care after implementing stringent emergency infection protocols. The study showed there was a dramatic increase in the time from symptom onset to first medical contact, an increase in the door-to-device time, and a delay in the time from cath lab arrival to device implantation.

“This was something we were talking about at our institution and it was something colleagues were talking about around the country,” said Nallamothu. “How do we maintain systems for emergency cardiovascular care when we’re being asked to shut down a lot of things? We thought [the Hong Kong paper] would be newsworthy and very helpful for clinical providers, so we made the decision to publish it. Even a few weeks to months later, we might have been more hesitant, largely because the message was starting to get out there.” 

There’s nothing worse than publishing something unlikely to be true. At the same time, it’s a bit of a wobbly bar because some information is needed. James de Lemos

Like Nallamothu, the Circulation editors brought in outside experts to help review research at an accelerated pace and met daily to consider articles for publication. For papers under consideration, reviewers were given 24 to 48 hours, an exceptionally rapid turnaround. Alongside those efforts, said de Lemos, they pioneered new content dedicated to the pandemic. In the earliest days of the pandemic, for example, when there was little information available, Senior Associate Editor Biykem Bozkurt, MD, PhD (Baylor College of Medicine, Houston), gathered a team of cardiologists to interview physicians on the front lines. In this way, doctors could relay information as to what they were seeing, such as the heightened risk of thromboembolic events with COVID-19. 

“We didn’t pretend this was verified scientific knowledge,” said de Lemos. “It was just anecdote. In the early phase, anecdote was better than nothing. These were trusted colleagues. It was just a trusted colleague telling you what they’ve seen and what to watch out for based on their experience. That was a big part of our focus early.”

Chandrashekhar said their journal received hundreds of COVID-19-related papers but they used a very fine filter for publication, leading to just four original research articles focused on COVID-19 published in 2020. One paper was the first to show that RV longitudinal strain was an important prognostic marker in patients with COVID-19. The journal also published one of the CMR studies suggestive of myocardial damage in recovered patients, but one which found that this wasn’t as prevalent as earlier reports had suggested. They also published another study indicating no signs of lasting cardiac damage on CMR imaging in athletes despite concerns circulating especially among professional and amateur sports organizations.

“This was our fear,” he said. “The early, frothy science had to settle down before we can really make any meaningful decisions. I’m glad we held off on publishing some papers because it misleads people. We’ve seen that with hydroxychloroquine and the story abut cardiac damage on CMR.” Although we’re now heading into year two of the pandemic, Chandrashekhar said he’s still receiving a high volume of papers and isn’t too impressed with the quality. “I would say there’s a lot of noise out there and very little light,” he said.     

More Than 10,000 Articles Published by May

In the review by Raynaud, Loupy, and colleagues, which was published January 4, 2021, in BMC Medical Research Methodology, the researchers identified 10,516 articles English and Chinese-language articles published as of May 2020, of which 7,468 were peer reviewed. This first-wave, said Loupy, was a critical period—the start of the pandemic—when clinicians had absolutely no experience with COVID-19 and knew nothing about patient management, potential therapeutics, or preventing transmission, among other daunting unknowns.

I think some of what was published early was poor quality. David Newby

Of the peer-reviewed publications, 4,190 (56.1%) were opinion pieces that did not contain any data or analysis. Just 1,109 (14.9%) articles were considered original studies with data. Case reports (9.3%), research letters (10.5%), reviews (8.5%), and systematic reviews (0.6%) made up the rest of the peer-reviewed publications.

Since February 2020, published articles related to COVID-19 increased from 203 articles per week to a whopping 1,645 articles published during 1 week in April 2020. Over the study period, articles without original data dominated the exponential growth of COVID-19 publications, with 262 published every week. There were, on average, 69 original studies published each week, followed by 49 research letters, 43 case reports, 40 reviews, and three systematic reviews.

“It was mainly people providing advice and their reaction to the pandemic,” said Loupy. “There were a lot of opinion pieces. Is it really science? I doubt that it is. Was it useful? Maybe, at some point, but it wasn’t a data-driven approach.”

The top five publishing countries were China, with 26% of all publications, followed by the United States (22.3%), the United Kingdom (8.8%), Italy (8.1%), and India (3.4%). The most-represented topics were infectious disease, epidemiology, global health, public health, and therapeutics, drugs, and medicines. However, researchers from nearly every field of medicine published articles related to COVID-19, including cardiology and internal medicine.

The critical appraisal focused solely on 713 clinical, peer-reviewed original articles comprised of observational and interventional studies. Numerous validated tools, such as the Cochrane Risk of Bias 2 and ROBIN-1 for evaluating bias in randomized and nonrandomized trials, respectively, were deployed to assess the quality of the studies.

That assessment concluded that very few of these original articles met high scientific standards, including the randomized and nonrandomized trials.

“We were really struck that there were very few papers with original data, and many—the vast majority of those papers—had bias,” said Loupy. “Methodological bias, small sample size, overstating conclusions, improperly designed studies, wrong statistics, and low-quality data. It was quite obvious. Many people knew about this, but ours is the only study to date to measure it in a very standardized manner.”

The Path Forward

In fairness, Newby also noted that much of the scientific world was unable to conduct normal research during the first few months of the pandemic, and as such, they turned to COVID-19. “For scientists, if they can’t do anything but COVID research, they’re going to do COVID research,” said Newby.

de Lemos added that all journals want to publish clinically relevant data that stands the test of time. The irony is that as more and more people are vaccinated and the virus is brought under control, COVID-19 papers will become less relevant to practicing physicians.

All research is contextual, Nallamothu noted: trends in submissions and publication are a product of the current time. “Now that we’re seeing a path out of this, we’re going to start seeing some reflective articles, such as this one [by Raynaud et al], about what did we learn about all of this,” he predicted.

To TCTMD, Loupy said their meta-research and critical appraisal project was a massive undertaking, with the 19-person group comprised of researchers with specialties in high-volume article analytics. However, getting the study published was a challenge, he said, because many journals saw it as a direct criticism of their processes, which the paper was not intended to do. “People really tried to help, but the lesson is that we need to be careful with the output of information because during a pandemic, agencies, like the [World Health Organization], and countries all over the world adapted their policies most of the time based on bad research,” he concluded.

Loupy said that he believes the vaccine studies have been well designed and that the initial rush to publication is subsiding. “I think we’re entering a period where we’re producing more and more good-quality papers,” he said. “That’s good news.”