Cited Studies on Biomarker Discoveries Prone to Exaggerating Effects

Studies are increasingly linking biomarkers to disease risk, treatment response, and disease progression. A recent literature review in the Journal of the American Medical Association [2011;305(21):2200-2207] indicates that some associations might be less robust than the initial studies suggest. Using the ISI Web of Science and MEDLINE databases, a team of researchers from the United States and Greece searched for studies on biomarker discoveries published in 24 prominent biomedical journals prior to 2010 and cited in the literature ≥400 times. One article per study was eligible for inclusion, and the article’s abstract had to report relative risk (RR), using metrics such as risk ratio, odds ratio, or hazard ratio. Randomized trials were excluded, based on an assumption that RR estimates were likely to refer to therapeutic effects instead of biomarker associations. Researchers identified 35 eligible articles, which were published in 10 journals between 1991 and 2006 (median, 1996) and received a median of 645 citations. Of the 35 biomarker associations, 11 concerned genetic risk factors, 11 involved blood proteins or other blood biomarkers, 6 assessed infectious agent biomarkers, and 7 were uncategorized. Articles associating biomarkers with cancer and cardiovascular disease were most prevalent, accounting for 14 and 12 studies, respectively. PubMed was searched to identify meta-analyses on each biomarker and associated health condition or outcome described in the 35 highly cited studies. If multiple meta-analyses were available, investigators chose the one that reviewed the most studies. With few exceptions, they excluded any meta-analysis that did not feature data from the corresponding highly cited study. Once studies were confirmed, the researchers compared RRs between the highly cited study, its corresponding meta-analysis, and the largest study referenced in the meta-analysis, which occasionally was the highly cited study. Of the 35 biomarker associations examined, the corresponding highly cited study reported a stronger effect size than the largest study for 30 (86%) of them. In 3 cases, the cited study and the largest study reported the same effect size, and in 2 cases, the largest study reported a stronger effect size. For 5 associations, the largest study and the one cited reached opposite conclusions regarding RR. In 20 cases, the estimated RR in the highly cited study was at least twice that reported in the largest study, and for 13 associations, the RR was 4 times higher in the highly cited study. Comparisons between the 35 highly cited studies and each one’s corresponding meta-analysis showed the meta-analysis reported a smaller effect in 29 cases. For 14 associations, the estimated RR in the highly cited study was more than double that reported in the meta-analysis, and for 7 associations, RR was more than quadruple in the cited study. Investigators said findings suggest “many of these highlighted associations” are exaggerated, with only 1 in 5 demonstrating a RR >1.37 per the largest studies. Possible explanations for the inflated results reported in the highly cited studies include their typical small sample, selective reporting, and, for approximately half, their earlier publication date. Despite apparent exaggerations, the authors said several biomarkers evaluated likely are associated with the indicated condition or outcome, even significantly in some cases. As examples, they pointed to Helicobacter pylori with gastric cancer and CFH with age-related macular degeneration. One concern is that overestimating the effect could have led to routine screening for the biomarkers in clinical practice without an accurate determination as to their cost-effectiveness. “Clinical biomarker use requires robust evidence and safeguards,” the authors said. They recommended readers be cautious when articles reference a single study but not a meta-analysis and suggested authors exercise more care when determining which sources to cite. Limitations of the study include lack of subjectivity in how studies were selected for inclusion, particularly the decision to restricting eligibility to studies citing RR in the abstract and having a related meta-analysis. It is also not certain these findings are applicable to less-cited reports on biomarker associations.