How do you feel about this alternative method for peer reviewing scientific papers?

[ZombieHorde]

Peer review under the microscope: one journal's experiment aims to change science vetting

snip

The peer review process has changed little from journal to journal for centuries, and Kravchenko was expecting that his most recent paper would get vetted through the same process that put his 80-odd earlier papers into print. This time, however, an e-mail from Nature invited Kravchenko to participate in a peer review experiment. Besides sending research articles to two or three anonymous reviewers, the journal was posting some papers on a Web site where any interested scientist could voice his or her opinions about the research, as long as the commenters revealed their identities.

snip

Opening the peer review process to the larger scientific community could have multiple benefits--scrutiny by a broader audience may give study authors more ideas for improving their research and catch more low-quality and fraudulent papers before they enter the scientific record. But posting papers before they're published might also open scientists' work to plagiarism or the possibility of being scooped by competing labs.

--------
More at link.
http://findarticles.com/p/articles/mi_m1200/is_25_170/ai_n17114210/pg_1?tag=artBody;col1

[metapunditedgy]

1. Sounds nice, but it takes a lot of time to review papers correctly, and

I doubt this scheme will be able to convince folks to invest that much time as hundreds of potential papers go by on the web. End result will be less careful reviewing. IMO, a review wiki could be instituted as a supplement to the current system, or as a "comment system" for published works.

[Arctic Dave]

2. "a review wiki"

This would be the problem I could see. Corporations would wiki bomb anything they didn't like or use it to hype their product.

[caraher]

3. Standard peer review is far from perfect, but the alternatives are not yet fully baked

One big problem is the motivation to do a solid review. It can be hard enough to get reviewers as it is.

It's also not quite true that "the process has changed little... for centuries." Consider just Einstein's own experience where very different standards and practices in Physical Review compared to other journals led to his boycotting Physical Review.

Maybe it all depends on what counts as a non-"little" change, but I think expectations have varied in important ways among fields, over time and from place to place.

[drmeow]

4. While the peer review process

definitely needs some revision, this is not the route to take. The "discussion papers" option mentioned in the article has some merit but this one is potentially problematic. Had I been asked, I would have refused.

A first step that I see as critical to making the review process better is to have completely blind reviews (where the reviewer does not know who wrote the paper). Some fields/journals already use this but others don't or use it in a way that does not completely mask the author. As a result, well known authors are more likely to get published than unknowns. Other things that would help advance science: publishing non-significant results (something NOT having an effect on something can be as important as something having an effect), editors being skeptical about negative comments about papers that challenge the status quo which are given poor reviews by "old school" scientists, and curbing reviewers requirements that the authors rewrite the paper in the reviewers image rather than that of the author.

[TZ]

5. I agree with your statement

Another problem with this is potentially opening up review of articles to scientists who AREN'T knowledgeable in the area.
There is so much specialization these days that even many with scientific backgrounds are not qualified to review certain articles really.
FWIW, a theoretical physicist is not a peer of a molecular biologist. I think doing this could cause a lot of problems.
And lets not forget that there are unfortunately people in the scientific community with political agendas.
Imagine an anti-vax quack reviewing a study on vaccine efficacy for instance.

[NNadir]

6. I agree, but I note that some quacks get "published" quite readily.

Ernie Sternglass of the infinite series of "strontium baby teeth" comes to mind and Randall Mills of the "hydrazino" come to mind immediately.

It is, of course, something of a good thing that this is so on some level, because it familarizes one with the "out there" kind of stuff.