ISCHEMIA Fracas: Amid Charges of Moving the Goalposts, Investigators Come Out Swinging

It’s going to take at least a year before the International Study of Comparative Health Effectiveness with Medical and Invasive Approaches (ISCHEMIA) study wraps up, but late-in-the-game changes to the trial have sparked a very public debate between ISCHEMIA investigators and their critics.

Some experts are now asking whether the contentious changes mean that this long-awaited trial won’t be able to answer key questions that have haunted the field since the COURAGE study first questioned the role of revascularization in stable coronary artery disease.

In a perspective published March 12, 2018, in Circulation: Cardiovascular Quality and Outcomes, a group from the United Kingdom accused the ISCHEMIA investigators of “moving the goalposts.” They were referring to the decision to alter the primary endpoint 5 years after the study’s launch to include not just cardiovascular death and MI—the study’s initial primary endpoint—but also resuscitated cardiac arrest, hospitalization for unstable angina, and hospitalization for heart failure.

In their perspective, Christopher Rajkumar, MBBS (Imperial College London, England), along with Sukhjinder Nijjer, MBBS, PhD, Graham Cole, MBBS, PhD, Rasha Al-Lamee, MBBS, and senior author Darrel Francis, MD (all from Imperial College London), point out that ISCHEMIA was designed around the primary endpoint of cardiovascular death and MI, a “hard” clinical outcome resistant to bias. Changing the endpoint now weakens the large National Heart, Lung, and Blood Institute (NHLBI) study, they say, encouraging physicians instead to focus on CVD death and MI when the study is finally completed.  

“We must learn to accept the results of trials even if they are not as we hoped,” they write. “True scientists do experiments for the potential to surprise.”

Needless to say, the accusations have stung the ISCHEMIA investigators, particularly since—they insist—no such thing occurred. The goalposts were planted firmly in the ground in 2011 when the trial was designed (with provisions for the primary endpoint’s expansion) and the grant application submitted to the NHLBI for funding.

“This trial has absolutely adhered to the highest, most rigorous clinical trial standards,” ISCHEMIA study chair Judith Hochman, MD (NYU Langone Medical Center, New York, NY), told TCTMD. “I think if you talk to anyone on our steering committee, or any of the investigators, they would verify that. It was concerning that the publication appeared to be attacking the trial.”     

ISCHEMIA was designed to determine the best treatment course in patients with stable coronary heart disease with moderate-to-severe ischemia on stress testing. Specifically, the study will test whether an invasive strategy of cardiac catheterization followed by coronary revascularization (PCI or CABG) plus optimal medical therapy provides incremental benefit over a conservative treatment strategy of optimal medical therapy alone.

The hope is that the trial will conclusively determine whether ischemia can single out those patients who will definitively benefit from revascularization from those who will do just as well with medical management. Earlier studies, including a nuclear substudy in COURAGE, have shown that PCI beats medical therapy for reducing ischemia, but whether that translates into hard endpoints avoided has been the million-dollar question in this fraught field.

The ISCHEMIA trial investigators have received National Institutes of Health (NIH) grants totaling nearly $100 million to conduct the study, but the possible cost savings are undeniable. Invasive angiography, PCI, and CABG are expensive, costing significantly more than medical therapy alone. Physicians, hospital administrators, healthcare economists—not to mention patients—would like some clarity as to whether more invasive and expensive treatments translate into meaningful clinical benefits, namely fewer MIs and better survival.

Launched in late 2011 and plagued by slow enrollment, the trial has finally reached its target, randomizing a full 5,179 patients to either coronary revascularization or optimal medical care. Full results are expected in 2020.

Not So Fast, Say ISCHEMIA Investigators

Shortly after the initial criticisms of the trial were published, Hochman and co-chair/principal investigator David Maron, MD (Stanford University School of Medicine), released their rebuttal explaining how and why the trial changed. They state that the grant application to the NHLBI was based on the current five-component primary endpoint, but that they sought and received approval to implement the study protocol with the primary endpoint of CVD death and MI. As part of the protocol, however, they agreed to convene an independent advisory panel to look at the possibility of a prespecified primary endpoint change—from CVD death and MI to the expanded five-component MACE—in the event it became necessary to preserve study power.

In May 2017, the independent panel reviewed aggregate outcome data and recommended the trial be expanded to the five-component MACE. The panel recommendation, which was independent of study leadership and the Data and Safety Monitoring Board (DSMB), was accepted by the NHLBI in June 2017. The decision to switch the primary endpoint to the five-component MACE was designated to occur before 75% of the final number of primary endpoint events accrued.  

Speaking during a recent American College of Cardiology (ACC) 2018 Scientific Session devoted to COURAGE, ISCHEMIA, and ORBITA, Maron addressed the change in the primary endpoint. When investigators designed the trial and projected the event rate—“as conservatively and as realistically as possible”—they hoped they would get it right, but had a backup plan just in case they didn’t, he explained.

“There’s always an overestimation of what the event rate is going to be,” said Maron, referring to the pitfalls of designing clinical trials. “So we wrote into the original protocol, now 8 years ago, that if the observed event rate falls short of the projected event rate, we would have a contingency plan to expand to a five-component event rate that would include the other three ‘softer’ endpoints as a way to preserve power.”

The ISCHEMIA trial originally planned to randomize 8,000 patients with stable ischemic heart disease, but this number was later reduced to between 5,000 and 6,000 patients. The slow recruitment has often been blamed on a lack of equipoise—bluntly put, too many physicians already believe they know the answer to the questions at the heart of the study. In a bid to boost enrollment, investigators posted a video on the Stanford University website featuring Maron in a tongue-in-cheek parody of Elvis Presley, pleading with colleagues through song: “It’s now or never, I ask each site/Enroll one patient before tonight.”

Hochman, who chaired the ACC session, stressed the endpoint change was not made post hoc, but rather as part of the study protocol.

“All five of those event [endpoints] were adjudicated from the very beginning by a blinded, independent adjudication committee because we knew we might need to switch,” she said. “It was a completely independent panel looking at blinded, aggregate data. We also prespecified that we would make this change after a certain number of projected events were accrued, not at the very end [of the trial]. We stuck to exactly what we prespecified at the beginning.”

When ClinicalTrials.gov was updated in January 2018 to reflect the change in protocol, an explanation stated that this was “consistent with the requirement to update this type of protocol change annually.”

Speaking with TCTMD, Francis praised both Hochman and Maron for “developing and executing an excellent clinical trial,” noting that the medical community have only themselves to blame for the low numbers. Ultimately, though, he believes the focus should remain on CVD death and MI. Although there likely will be fewer events than hoped, clinicians can place ISCHEMIA in the context of other stenting trials, as they’ve done historically with other research.

Francis said their group only discovered the change in the primary endpoint when they were researching “faith healing” and “subtraction anxiety,” two biases that can creep into unblinded clinical trials.

At the time they were drafting their perspective, there was no public information that the primary endpoint had been changed in 2017, nor was there any information that the NHLBI had originally approved and funded the study based on the five-component MACE. Moreover, there was nothing in the public domain to suggest the study would revert back to the expanded endpoint in the event there’d be an insufficient number of clinical events, said Francis.

If ‘Soft’ and ‘Hard’ Events Go in Different Directions

One of the biggest questions raised by the Imperial College group, which includes several researchers from the sham-controlled ORBITA trial comparing PCI and medical therapy in patients with stable CAD, is whether the introduction of resuscitated cardiac arrest, unstable angina, or heart failure as part of the primary endpoint weakens the study, particularly given that these endpoints are softer, more subjective, and susceptible to bias.

In their perspective, the group states that CVD death and MI are “largely objective, dichotomous events” but hospitalization for unstable angina and heart failure could influence the study results since physicians and patients will both be aware of the treatment course. “Unblinded trials of symptoms are as harmful for the study of procedural interventions as they are for the study of medications,” they write.

Sanjay Kaul, MD (Cedars-Sinai Medical Center, Los Angeles, CA), who is not involved in ISCHEMIA, said things will get interesting if adding PCI to medical therapy reduces the risk of the five-component primary endpoint but the benefit is driven by reductions in unstable angina and heart failure. In that case, he said, he’ll have less confidence in the results. Additionally, he will be keeping a close eye on the revised key secondary endpoint—CVD death and MI—to see if it aligns with the primary endpoint.

“If you have situation where the five-component endpoint is positive but CVD death and MI is null, or moves in the wrong direction, then the results will certainly have less credibility for me,” said Kaul.

For Venkatesh Murthy, MD (University of Michigan, Ann Arbor), ISCHEMIA is one of the most important trials going on in cardiology, possibly the most important of the last 10 years. He praised the study leadership, particularly given their dedication to a large trial asking a clinical question that many interventional cardiologists believe in their heart of hearts is a settled issue.

ISCHEMIA, he said, is likely the last chance to determine the true value of revascularization in the setting of stable ischemic heart disease.

“We have a lot of drug, stent, and device trials,” including large comparative studies between different devices, said Murthy. “But it’s very unlikely we’re going to get another ISCHEMIA trial, or anything vaguely like it. This trial should hopefully stand the test of time, so if it doesn’t give us a clear answer, I think it’ll be tragic.”

Like Kaul, Murthy said he doesn’t believe the study will be conclusive if there is only a benefit on unstable angina/heart failure hospitalizations. “But yet we don’t know,” he said. “We’re all speculating here. It might turn out that in 2020 when results are presented, a minimum of 18 months from now, that all the endpoints line up in the same direction and then nobody will say this isn’t definitive.”

William Boden, MD (VA New England Healthcare System, Boston, MA), one of the study’s principal investigators, told TCTMD that concern about potential biases creeping into the study is a fair argument.

“Death and MI are pretty incontrovertible and are rather unambiguous, even in an unblinded trial,” he said. “What ORBITA did show us is that when you begin to add hospitalization for ACS and heart failure, those endpoints have the potential to engender some bias on the part of the physician-investigator who obviously knows the [type of] treatment to which that patient has been assigned.”

He pointed to comparisons between COURAGE and FAME-2, two similar studies that compared PCI plus optimal medical therapy versus optimal medical therapy alone in patients with stable coronary artery disease. Whereas COURAGE used death from any cause and nonfatal MI as the primary endpoint, FAME-2 used all-cause mortality, MI, and urgent revascularization. 

That “third” endpoint in FAME-2, said Boden, “is subject to inadvertent treatment bias because if patients had angina, and you know they got the medical therapy assignment, then your threshold for potentially hospitalizing the patient, and potentially crossing them over, will be lower. I think that’s an inescapable criticism to some degree about the inclusion of these so-called ‘softer’ endpoints as part of the primary composite endpoint.”

Likewise, Kaul said the concerns of potential bias are legitimate, telling TCTMD that hard endpoints are preferable in trials designed to resolve uncertainties in clinical practice. Whether PCI offers a substantial, incremental advantage over an initial strategy of medical therapy in stable coronary disease is one of those major uncertainties, he said.  

Still, Kaul believes it’s good clinical trial practice to anticipate a lower-than-expected accrual of hard clinical events and to allow for a contingency plan. With a shortage of hard events accrued, there are typically two options: extend the follow-up or increase the number of patients enrolled.

“For an NIH-sponsored trial with limited resources, both of these are, in my mind, not desirable,” he said. “So before unblinding, I would argue it’s not unreasonable to change the endpoint based on the prespecified contingency plan provided the new components of the composite endpoint can be accurately ascertained. That’s key: an accurate ascertainment and adjudication based on objective criteria to minimize bias.”

Sripal Bangalore, MD (NYU Langone Medical Center), the lead investigator of the ISCHEMIA ancillary study of patients with advanced chronic kidney disease, said clinicians and researchers can always make a case for or against any of the “softer” endpoints. From a patient’s perspective, however, as well as from a resource utilization standpoint, hospitalizations for unstable angina and heart failure are important outcome measures, he said.

Objectively Defining Subjective Endpoints

Speaking with TCTMD, Hochman disagreed with the characterization of unstable angina and heart failure hospitalizations as weaker endpoints, at least in terms of how they are being used in ISCHEMIA. The adjudication of clinical outcomes in the trial has been “incredibly strict,” and researchers have made every effort to minimize or eliminate reporting bias, she said.

All clinical outcomes will be meticulously defined, reviewed, and confirmed, and “I don’t think anybody is going to say that they’re soft,” said Hochman. For example, unstable angina will be defined by significant changes on an ECG confirmed by a core lab, and heart failure hospitalizations will also be based on objective criteria. The researchers plan to publish a paper shortly outlining the rationale and study design, and this will include detailed explanations on the objective criteria used to define the expanded clinical endpoints.

While the careful adjudication of clinical events is critical and would be expected with excellent trial leadership, Murthy said some things can’t be captured. “If doctor gets a phone call about a patient who just had bypass surgery, and they describe an atypical, mild symptom that’s not exertional and may not be really concerning for coronary disease, there’s a chance the person doesn’t even get sent to the hospital,” he said.

The tendency for physicians to take action given what they know is difficult to guard against. “You can never fix that,” said Murthy. “The documentation for those phone calls is often sparse, and you don’t know what the blood work would have shown because it was never drawn. You don’t know what the ECG would have shown, because it was never taken.”

Gregg Stone, MD (NewYork-Presbyterian/Columbia University Irving Medical Center, New York, NY), another ISCHEMIA principal investigator, said the best endpoint in any cardiovascular trial is all-cause mortality, as it’s easy to adjudicate and confirm and least subject to bias. Even differentiating between CVD and non-CVD death can be surprisingly nuanced, he said.  

“All other endpoints, including MI, heart failure, rehospitalization, and revascularization are highly dependent on definition and are more prone to ascertainment and other biases,” he said. “And yes, in general, double-blind or sham-controlled trials are ideal to minimize bias, but in a trial that includes bypass surgery this is not possible.”

Stone added that many steps can be taken to minimize bias and optimize discrimination in open-label trials, including protocol requirements, prespecified definitions and endpoints with sensitivity analyses, complete data monitoring with event ascertainment (such as with routine assessment of postprocedure biomarkers and hospital/outpatient notes), independent event adjudication, core laboratory assessment, and so on.

‘Holy Cow, They’ve Changed the Primary Endpoint!’  

Boden said the ISCHEMIA leadership has been stung by the criticisms from the UK group. He did agree that investigators could have done a better job explaining the decision and letting the cardiology community know about the change in the primary endpoint when it was made in 2017. 

“It wasn’t really until January 2018 that the ClinicalTrials.gov ISCHEMIA website was updated to reflect the change,” said Boden. “I think this is what Darrel Francis and the group reacted to. They looked and said, ‘Holy cow, they’ve changed the primary endpoint; this is post hoc, and all done very hastily near the end of the trial.’ The optics this created was that we had something to hide, or we were being less than forthcoming or transparent about the primary endpoint of the study, when in fact it’s quite the contrary.” 

The decision to change the endpoint, he said, had been discussed extensively by the executive committee, the steering committee, the NHLBI, and DSMB, and it was not something they undertook lightly. 

To TCTMD, Hochman said ISCHEMIA is larger than several other studies published to date, including COURAGE, which randomized just 2,287 patients. Given that their study is more than twice the size, they still hope to provide meaningful results on the outcome of CVD death and MI, which remains a major secondary clinical endpoint.

“It’s of great importance to us,” said Hochman. “We had hoped to achieve enough power for it to remain the primary endpoint, but we basically had to make a decision. As the prespecified analysis showed, we ran the risk of being underpowered. We had a choice of continuing an underpowered trial or reverting to the outcome that was peer-reviewed by an independent panel for the NHLBI and on the basis in which they funded us.”   

Boden noted there is still another 16 months of follow-up remaining in the trial, with the last patient completing follow-up in June 2019. While the recommendation from the independent panel to expand the endpoint suggests the study is underpowered for CVD death and MI, the study might still get there.

“I’ve certainly been involved in trials where as you go deeper into follow-up, the endpoints accrue and you have more of them,” he said. “That’s why you see in so many of these trials the curves separating late. The event curves might separate later in follow-up as events accrue, so I don’t know if we or anyone can say there will be sufficient power.”

That leaves open the question of whether tens of millions of dollars have been spent to get answers that won’t change the minds of physicians who weren’t ever invested in ISCHEMIA in the first place.

Francis, who has been outspoken with his criticisms and questions about the endpoint change on Twitter, told TCTMD that he does not believe the slow enrollment and low event rate can be blamed on the trial leadership, but rather on participating physicians and hospitals who failed to randomize patients into the trial. He said if ISCHEMIA was going to be underpowered for hard endpoints, trial leadership should have simply said so, rather than opting to go with the expanded five-component MACE endpoint that has led to the current controversy.

“If we interventional cardiologists have double-crossed them by not recruiting, it is our fault and not theirs,” said Francis. “They could extend the trial. If there is no money for that, again they should simply say so. That’s not their fault either.”