Welcome to DU!
The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards.
Join the community:
Create a free account
Support DU (and get rid of ads!):
Become a Star Member
Latest Breaking News
General Discussion
The DU Lounge
All Forums
Issue Forums
Culture Forums
Alliance Forums
Region Forums
Support Forums
Help & Search
Impact factor predicts unreliability of research papers
http://bjoern.brembs.net/comment-n815.html
Impact factor predicts unreliability of research papers
Last week, we've already seen that the most prominent way of ranking scholarly journals, Thomson Reuters' Impact Factor (IF), isn't a very good measure for predicting how many citations your scientific paper will attract. Instead, there is evidence that IF is much better at predicting the chance that your paper might get retracted.
Now, I've just been sent a paper (subscription required) which provides evidence that the reliability of some research papers correlates negatively with journal IF. In other words, the higher the journal's IF in which the paper was published, the less reliable the research is.
<snip>
Taken together, this study provides some evidence for one of the potential mechanisms underlying the very strong correlation between IF and retractions we've seen before: authors are more likely to publish unreliable data with predominantly overestimated effect sizes in high-IF journals. Importantly, this constitutes a mechanism which cannot be explained by high-IF journals being more closely scrutinized than low-IF journals. Instead, it suggests that at least a portion of the retractions in high-IF journals is due to the studies published there being more likely to be flawed than studies in low-IF journals.
In the words of the authors:
Impact factor predicts unreliability of research papers
Last week, we've already seen that the most prominent way of ranking scholarly journals, Thomson Reuters' Impact Factor (IF), isn't a very good measure for predicting how many citations your scientific paper will attract. Instead, there is evidence that IF is much better at predicting the chance that your paper might get retracted.
Now, I've just been sent a paper (subscription required) which provides evidence that the reliability of some research papers correlates negatively with journal IF. In other words, the higher the journal's IF in which the paper was published, the less reliable the research is.
<snip>
Taken together, this study provides some evidence for one of the potential mechanisms underlying the very strong correlation between IF and retractions we've seen before: authors are more likely to publish unreliable data with predominantly overestimated effect sizes in high-IF journals. Importantly, this constitutes a mechanism which cannot be explained by high-IF journals being more closely scrutinized than low-IF journals. Instead, it suggests that at least a portion of the retractions in high-IF journals is due to the studies published there being more likely to be flawed than studies in low-IF journals.
In the words of the authors:
Our results indicate that genetic association studies published in journals with a high impact factor are more likely to provide an overestimate of the true effect size. This is likely to be in part due to the small sample sizes used and the correspondingly low statistical power that characterizes these studies. Initial reports of genetic association published in journals with a high impact factor should therefore be treated with particular caution. However, although we cannot necessarily generalize our findings to other research domains, there are no particular reasons to expect that genetic association studies are unique in this respect.