Yet Another Study Underscores Drawbacks of Risk-Adjusted Mortality as a Measure of PCI Quality

Adding fodder to the debate over the best way to publicly report PCI outcomes, a large registry study shows that risk-standardized in-hospital mortality after the procedure varies substantially across interventional cardiologists in the United States, even among those meeting recommended minimum procedure volumes.

Moreover, an operator’s status based on having a high, low, or average mortality rate is generally not consistent from year to year, researchers led by Jacob Doll, MD (VA Puget Sound Health Care System, Seattle, WA), report in a study published in the April 10, 2017, issue of JACC: Cardiovascular Interventions.

Having a higher-than-average mortality rate did not seem to be influenced strongly by case mix or procedure characteristics, they note.

“Although this suggests that operator [risk-standardized mortality rate] may be useful to identify opportunities for quality improvement, its poor year-to-year stability and concerns about the adverse effects of public reporting limit its use as a sole performance measure,” Doll et al write. They add, however, that “operator PCI mortality could be one component of quality assessment if considered in the context of other outcomes and process measures.”

Commenting for TCTMD, Frederic Resnic, MD (Lahey Hospital & Medical Center, Burlington, MA), who was not involved in the analysis, also endorsed a more holistic assessment of PCI quality that includes factors beyond risk-adjusted mortality, such as appropriateness of the procedure and the willingness of physicians to treat the sickest patients. He said this study “adds to some accumulating information that risk-adjusted mortality . . . at the PCI operator level has a lot of limitations and issues.”

Of “significant concern” to Resnic is the fact that whether or not individual physicians were identified as outliers depended largely on what patient population was examined, which was a pattern also seen in another recent study. What that suggests, he said, is that operators considered major outliers when all patients are included, but not when high-risk patients are excluded, are outliers only because of the high-risk cases they perform. If the risk-adjustment models cannot be trusted to provide a fair comparison of PCI quality across operators regardless of case mix, then physicians may become reluctant to treat the highest-risk patients for fear of skewing their report cards, he said.

“The problem and paradox here is that it’s in those populations of higher-risk cases where the greatest benefit to the patient exists,” Resnic pointed out.

Overall, “we don’t have a good handle on how to monitor, inform, and improve quality in PCI using these large registries,” he said. “Operator risk-adjusted mortality rate . . . seems to be inadequate as a sole measure of quality at the operator level—and I think also at the institutional level—and you have to complement it with a bunch of other factors and knowledge to understand what is the true quality.”

Instability in Outlier Status

Prior work in this arena suggest that the interventional community supports the concept of publicly reporting PCI outcomes to increase transparency and aid in quality improvement efforts, but many practicing physicians have concerns about how it’s executed.

Previous research has shown that higher-volume operators achieve lower mortality rates, but there is little information on variability in PCI mortality among interventional cardiologists meeting the recommended minimum number of procedures to maintain clinical competence (50 cases annually).

Doll et al explored that issue using the National Cardiovascular Data Registry (NCDR) CathPCI Registry sponsored by the American College of Cardiology and Society for Cardiovascular Angiography and Interventions. After excluding 63% of operators because of low case volumes, the analysis included 2,352,174 PCIs performed at 1,373 hospitals by 3,760 interventional cardiologists between October 2009 and September 2014. Operators performed a median of 103 PCIs per year, with an average in-hospital mortality rate of 1.5%.

A total of 242 operators (6.5%) had mortality rates greater than two standard deviations (SDs) above the mean, and 156 (4.1%) had rates greater than two SDs below the mean. Observed mortality rates were 2.2% in the high outliers, 1.5% in the nonoutliers, and 0.4% in the low outliers. Corresponding risk-standardized values were 2.7%, 1.5%, and 0.5%, respectively.

Outlier status was not stable from year to year, however. Physicians deemed high outliers for the entire study period were only classified as high outliers for an average of 1.53 out of 5 years. In fact, some overall high outliers were not considered such for any individual year.

Muddying the waters even further, only 108 operators were classified as high outliers both in the main analysis and in an analysis of a lower-risk population that excluded patients who underwent emergency/salvage PCI or had cardiogenic shock or cardiac arrest within 24 hours of presentation. Another 266 physicians were high outliers in only one of the two analyses.

“Our study shows that these operator classifications are dependent on the patient population selected for evaluation,” the authors write.

How to Measure PCI Quality

Doll et al state that PCI mortality at the operator level has limitations as an isolated measure of quality stemming from low overall mortality rates, the year-to-year instability of the metric, the fact that most deaths are not related to the procedure itself, and the potential for risk-averse behavior, which has been demonstrated in prior studies. Thus, an assessment of PCI quality should include metrics related to processes and structure at the hospital level, they say.

In an accompanying editorial, Michael McDaniel, MD (Emory University, Atlanta, GA), questions why in-hospital risk-adjusted mortality is being used to assess PCI quality if it is a poor indicator of quality.

McDaniel points out that the major professional societies make recommendations that contrast with the NCDR’s risk-adjusted mortality metric by advising exclusion of certain high-risk patients, such as those with cardiogenic shock or out-of-hospital cardiac arrest and those who have been turned down for surgery.

“Despite these concerns, high-risk populations are included in the NCDR risk-adjusted mortality metric,” he writes.

“Until we have disease-based registries that capture all patients (not just patients with PCI procedures), the NCDR’s in-hospital risk-adjusted mortality should not be used in value-based purchasing or public reporting unless patients with cardiac arrest, cardiogenic shock, and surgical turndowns are excluded to minimize risk aversion,” McDaniel concludes. “Perhaps if risk-adjusted mortality was measured on the basis of clinical presentation, included all patients regardless of treatment strategy, and evaluated over longer periods of time rather than in-hospital, then risk-adjusted mortality would better inform about the quality of care physicians and hospitals provide for patients with coronary artery disease.”

To TCTMD, Resnic stated that risk-adjusted mortality is here to stay as part of an assessment of PCI quality, because it wouldn’t be acceptable to patients or public health officials to stop tracking it.

“It’s better to continue to point out the weaknesses of mortality as a sole measure and say that the appropriate thing to do is to add to it all of the features that we can reliably calculate that are validated that together give us a much better picture,” he said.