As Pragmatic Trials Gain Steam, Beware the Pitfalls of Claims Data

Efforts are underway to make clinical trials more affordable, and one way to do that is to tap into existing data sources to tally adverse event rates during regular patient follow-up. But a new study shows that relying on billing claims instead of physician adjudication may miss some events—particularly when it comes to bleeding—and underscores the need for caution when considering that approach.

In a secondary analysis of the TRANSLATE-ACS study, which followed patients with acute MI who underwent PCI and received treatment with an ADP-receptor inhibitor, there was moderate correlation between claims data and physician adjudication for detecting recurrent MI or stroke through 1 year of follow-up, according to lead author Patricia Guimarães, MD (Duke University, Durham, NC), and colleagues.

But for overall bleeding, and especially for GUSTO moderate or severe bleeding, correlation was poor, they report in a study published recently in JAMA Cardiology.

“While medical claims may be a reasonable resource to assess MI and stroke outcomes, caution is still needed,” they say. “Medical claims have limited accuracy in identifying bleeding events, which suggests the need for an alternative approach to ensure good safety surveillance in cardiovascular studies.”

Tracy Wang, MD (Duke University), senior author of the paper and co-principal investigator of TRANSLATE-ACS, noted that there is growing interest in pragmatic clinical trials that can be performed more quickly and at lower cost while preserving evidence quality. Researchers have begun exploring use of existing data sources—like billing claims—to reduce the cost of patient follow-up, but there are some questions about the accuracy of that information, she told TCTMD. “If it’s not perfect, and we don’t think that it’s going to be perfect, then how can we strengthen our methodology in making sure that we’re getting at the truth when we’re looking at the claims datasets?”

To start answering that question, the investigators turned to TRANSLATE-ACS, a multicenter, longitudinal study of 12,365 patients with acute MI enrolled at 233 US hospitals. During 1 year of follow-up, rehospitalizations for recurrent MI, stroke, or bleeding were identified by looking at diagnosis and procedural codes in medical claims forms and through physician adjudication of medical records.

In general, event rates stemming from claims data were lower than those resulting from physician adjudication, although the discrepancy was mitigated when all of the diagnosis codes on a hospital bill—and not just the primary code—were included in the analysis.

Considering all codes, claims data provided lower 1-year rates of MI (4.3% vs 4.7%) and any bleeding (4.7% vs 5.4%) compared with physician adjudication; the stroke rate was 0.9% with each source.

The correlation between rates derived from billing claims data and physician adjudication was deemed moderate for MI (κ statistic 0.76) and stroke (κ statistic 0.55), but poor for any bleeding (κ statistic 0.24) and moderate or severe GUSTO bleeding (κ statistic 0.15). Patient-level analyses demonstrated somewhat better agreement between the two data sources.

Wang said the findings both reassured that there was “fairly good” agreement between claims data and physician adjudication for MI and stroke and pointed to a need for improvement in assessing bleeding. “Trying to use billing codes to find a bleeding event is problematic here,” she said.

In this era, we still need to do a little bit more due diligence in making sure that we’re not fully relying on billed data. Tracy Wang

Overall, physician adjudication leads to higher event rates, “so I think that gives you a sense that even though your accuracy for some of these events is pretty good [with billing claims], you’re probably still slightly under-detecting events,” she said.

Wang said that looking for ways to use billing data in the setting of pragmatic clinical trials remains worthwhile, however. She pointed out that even though events might be tallied differently, the conclusions of any particular study would likely remain the same. “I think the directionality of the results will very likely be preserved just using bill data, and that’s even true for bleeding.”

In an accompanying editor’s note, Gregg Fonarow, MD (University of California, Los Angeles), and Clyde Yancy, MD (Northwestern Feinberg School of Medicine, Chicago, IL), said the results of the study “should serve as a wake-up call.”

“Despite the momentum toward the use of less expensive, pragmatic trials and the financial attractiveness of alternative events identification and classification methods, using claims data alone for study follow-up and event adjudication may not offer, at least currently, the accuracy necessary to replace traditional approaches,” they argue. “Caution is needed when interpreting any study that uses these alternative methods.”

Wang did not disagree, saying that “in this era, we still need to do a little bit more due diligence in making sure that we’re not fully relying on billed data.”

But, she added, “I think this is a really exciting period of time where we can really think about doing pragmatic trials, and we should do that because the days of doing [trials costing several hundreds of millions of dollars] is really kind of behind us at this point. We can’t afford to do that, and we shouldn’t have to do that. But I think [what] we need to do now is focus on how we can make sure that our data can be accurately used for trial assessment.”