Psychology Secrets: Most Psychology Studies Are College Student Biased

[bananas]

Recently, someone demanded that I "prove" something by providing a link to a study in a peer-reviewed journal. I had to explain that most technical information never appears in peer-reviewed journals. So the first paragraph in this article caught my attention.

http://psychcentral.com/blog/archives/2010/08/26/psychology-secrets-most-psychology-studies-are-college-student-biased/

Psychology Secrets: Most Psychology Studies Are College Student Biased
By John M Grohol PsyD

Psychology, like most professions, holds many little secrets. They’re well known and usually accepted amongst the profession itself, but known to few “outsiders” or even journalists — whose job it is to not only report research findings, but put them into some sort of context.

One of those secrets is that most psychology research done in the U.S. is consistently done primarily on college students — specifically, undergraduate students taking a psychology course. It’s been this way for the better part of 50 years.

But are undergraduate college students studying at a U.S. university representative of the population in America? In the world? Can we honestly generalize from such un-representative samples and make broad claims about all human behavior (a trait of exaggeration fairly commonplace made by researchers in these kinds of studies).

These questions were raised by a group of Canadian researchers writing in Behavioral and Brain Sciences journal last month, as noted by Anand Giridharadas in an article yesterday in The New York Times:

...

http://www.nytimes.com/2010/08/26/world/americas/26iht-currents.html

A Weird Way of Thinking Has Prevailed Worldwide
By ANAND GIRIDHARADAS
Published: August 25, 2010

CORTES, CANADA — Imagine a country whose inhabitants eat human flesh, wear only pink hats to sleep and banish children into the forest to raise themselves until adulthood.

Now imagine that this country dominates the study of psychology worldwide. Its universities have the best facilities, which draw the best scholars, who write the best papers. Their research subjects are the flesh-eating, pink-hat-wearing, forest-reared locals.

When these psychologists write about their own country, all is well. But things deteriorate when they generalize about human nature.

They view behaviors that are globally commonplace — say, vegetarianism — as deviant. Human nature, as they define it, reflects little of the actual diversity of humankind.

...

The paper itself is titled "The weirdest people in the world?":

http://journals.cambridge.org/action/displayFulltext?type=6&fid=7825834&jid=BBS&volumeId=33&issueId=2-3&aid=7825833&fulltextType=RA&fileId=S0140525X0999152X

The weirdest people in the world?

...

Abstract

Behavioral scientists routinely publish broad claims about human psychology and behavior in the world's top journals based on samples drawn entirely from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies. Researchers – often implicitly – assume that either there is little variation across human populations, or that these “standard subjects” are as representative of the species as any other population. Are these assumptions justified? Here, our review of the comparative database from across the behavioral sciences suggests both that there is substantial variability in experimental results across populations and that WEIRD subjects are particularly unusual compared with the rest of the species – frequent outliers. The domains reviewed include visual perception, fairness, cooperation, spatial reasoning, categorization and inferential induction, moral reasoning, reasoning styles, self-concepts and related motivations, and the heritability of IQ. The findings suggest that members of WEIRD societies, including young children, are among the least representative populations one could find for generalizing about humans. Many of these findings involve domains that are associated with fundamental aspects of psychology, motivation, and behavior – hence, there are no obvious a priori grounds for claiming that a particular behavioral phenomenon is universal based on sampling from a single subpopulation. Overall, these empirical patterns suggests that we need to be less cavalier in addressing questions of human nature on the basis of data drawn from this particularly thin, and rather unusual, slice of humanity. We close by proposing ways to structurally re-organize the behavioral sciences to best tackle these challenges.

...

[MineralMan]

1. I've always maintained that Psychology is not a science.

Its results are not repeatable in any sort of reliable way. One cannot predict human behavior like one can predict things in other disciplines. Why? Because there are far too many variables involved in any experiment. Every human being is different, psychologically, from every other human being.

Shakespeare was as good a psychologist as anyone. Was he a scientist?

[fasttense]

2. There are many repeatable and reliable psychological studies.

In natural settings it is difficult to predict individual human behavior, but in controlled environments and in large statistical groups, you would be surprised at how predictable humans can be.

Some of that information has been put to good use. How do you think lie detector tests were developed? From experiments on human response to stress and lying.

Tests from IQ tests to SAT/ACT were developed by people studying psychology. Many useful personality tests have been developed to identify psychological problems.

The next time you watch an ad on TV, remember the study of psychology is responsible for many of the techniques and methods used in the commercial (ok, so that's a reason not to like psychology).

[MineralMan]

4. Polygraphs are not legally admissible evidence.

There's a good reason for that. They are not scientific, nor are they particularly accurate. Again, when you attempt to apply psychological "facts" to individuals, the results are not reproducible.

In science, if the same experiment produces different results on different occasions, the theory is disproved. Scientific theories can be falsified. If a theory is falsified, it is false.

Psychological theories are falsified all the time. I used to demonstrate that to psychology professors by deliberately doing something other than what was expected. I knew the theory, and simply falsified it when tested. You see, that capability in individuals is the single thing that falsifies most psychological theories.

Psychology deals in probabilities, not reproducible results. It is not science.

[Orsino]

5. Yes, its results are repeatable.

That human behavior is more complex than most other phenomena studied by science doesn't make that study unscientific.

Just more challenging.

[Taitertots]

6. Have you ever taken a college Psychology class?

I highly doubt it considering you think Shakespeare was a good psychologist.

Science or not, much of the results are absolutely valid. It is really a pedantic argument.

[RaleighNCDUer]

3. Funny, I've never heard of any study using college students as their sample

that did not admit up front that it used college students as its sample.

It is de-rigeur to say up front who the testing represents. Of course there may be a certain bias in those results because college students are disproportionately better educated and intelligent than the general populace - that's WHY they are in college. But any researcher worth his grant will use the student sampling to gather base data, and then, upon drawing conclusions from that, use his conclusions to get further funding to expand the study to the general populace.

May I say, "duh".

[Igel]

7. That, however, isn't the point.

In fact, it rather makes the point.

People who read the journals and see that the papers describe their subjects tend to be people in the field. People who read dozens of such papers and able to spot the generalization are almost surely in the field.

Newspaper writers, science reporters, etc., etc. talk about the methods and conclusions and only mention the subjects if it's appropriate. You study the reaction of people to images of black vs white youth, it's pertinent if the people involved are youth or 40 years old or seniors; if they're black or white. You study how people react in tests of honesty or altruism, the assumption is that black or white doesn't matter, and age is immaterial.

I've done a bit of psycholinguistics. We know a lot of about how college students process language. Mostly English, but also others. Still mostly college students.

I'm also a Slavist. It's simply amazing how many papers that relied on surveys didn't describe their subjects explicitly. You talk to the researchers, and it turns out that the cutting-edge work is done primarily on 60-something professors and lots of graduate students. Hardly a random sample of Russian, Czech, or Serbian speakers.