Live by statistics, die by statistics
Pretty interesting result -- scientists doing some science on themselves 
Now a paper has come out that ought to make some psychologists, who use that p value criterion a lot in their work, feel a little concerned. The researchers analyzed the distribution of reported p values in 3 well-regarded journals in experimental psychology, and described the pattern.
The circles represent the actual distribution of p values in the published papers. Remember, 0.05 is the arbitrarily determined standard for significance; you don’t get accepted for publication if your observations don’t rise to that level.
Notice that unusual and gigantic hump in the distribution just below 0.05? Uh-oh.
I repeat, uh-oh. That looks like about half the papers that report p values just under 0.05 may have benefited from a little ‘adjustment’.
What that implies is that investigators whose work reaches only marginal statistical significance are scrambling to nudge their numbers below the 0.05 level. It’s not necessarily likely that they’re actually making up data, but there could be a sneakier bias: oh, we almost meet the criterion, let’s add a few more subjects and see if we can get it there. Oh, those data points are weird outliers, let’s throw them out. Oh, our initial parameter of interest didn’t meet the criterion, but this other incidental observation did, so let’s report one and not bother with the other.
http://scienceblogs.com/pharyngula/2012/08/13/live-by-statistics-die-by-statistics/
Notice that if you are reading a bunch of science papers reporting p-values around 0.5%, the implication is that about 1/20 of those papers got a spurious result! The only question is, which ones...
Another fun fact about the famous "0.05 pval threshold": Suppose you are running an algorithm (like decision tree model training) that does *many* p-value tests. For example, training a single interior tree node split can easily involve thousands of such tests on a large data set, each of which has a 5% chance of being spurious, not "true." A pval threshold of 0.05 is a common parameter in such training algorithms, and if you run the numbers on this, you find that the odds of your chosen split being "truly significant" can be a lot less than 95%. An adjustment such as the Bonferroni adjustment is often employed to compensate for this problem.
= new reply since forum marked as read
Nice, clear, brief, jargon-free.
if it was a thing like "we aren't going to publish unless it's < 0.05" then I would expect to see all the dots to the left of 0.05 being measurably higher than the black line of prediction.
As Myers points out, there are a lot of non-deliberate ways this might be happening, but that are technically forms of cooking the data. One thing that could be happening is that you collect some data. You find you didn't quite make 0.05. you collect some more, and retest, and keep doing that. There isn't really anything wrong with that, however it happens that this injects a kind of experimental bias, because each time you run the experiment, you are increasing your odds of getting the p-value you want. The Bonferroni adjustment is meant to try and correct for that. If you're *really* being a stickler, you would apply the Bonferroni adjustment to account for how many times you ran your experiment before you reported your final p-value.
To give a better answer, I'd have to read more about how the authors collected *their* data
