Posted by Sam Fenny - Memes and headline comments by David Icke Posted on 22 May 2024

Are a Large Percentage of Recent Psychology Studies Flawed?

How do psychologists get data to test their theories? The traditional method was to recruit a sample of undergraduate students from their university. While this was very convenient, it had one obvious disadvantage: undergraduate students aren’t necessarily representative of the population – let alone all humanity.

In recent years, therefore, psychologists have turned to online survey platforms – the most popular of which is Amazon’s Mechanical Turk. How does it work? Individuals sign up to be ‘MTurk workers’ and can then be recruited to participate in academic studies. They are paid a small fee each time they participate in one. Specifically, they are paid a minimum of $0.01 per ‘assignment’, with a typical study involving many ‘assignments’.

MTurk has become extremely popular among psychologists and other academics. Over 40% of the articles in some journals are based on MTurk data, so we’re talking about thousands of studies. The problem is that most of these studies may be flawed.

Why?

A growing body of evidence indicates that MTurk data is of very low quality, due to the high percentage of MTurk workers who are careless responders. By ‘high percentage’ I mean upwards of 75%. Careless responders may click options randomly, or they may engage in what’s called ‘straightlining’ where they click the first option that appears for each successive question. Both types of responding yield data that is worthless.

In a new preprint, Cameron Kay provides a particularly vivid illustration of the problems with MTurk. His methodology was simple: he recruited a sample of MTurk workers and gave them 27 ‘semantic antonyms’. These are pairs of items that ought to yield opposite answers: for example, “I am an extravert” and “I am an introvert”, or “I talk a lot” and “I rarely talk”.

If most respondents are paying attention and taking the study seriously, the pairs of items will be negatively correlated. There’ll be a tendency for people who agree with “I am an extravert” to disagree with “I am an introvert”. By contrast, if most respondents are straightlining, the pairs of items will be positively correlated. And if most respondents are clicking randomly, the pairs of items won’t be correlated at all.

What did Kay find? 26 out of 27 pairs of items were positively correlated. In other words, there was a tendency for respondents who agreed with “I am an extravert” to agree with “I am an introvert” – complete nonsense. This is shown in the chart below.

Read More: Are a Large Percentage of Recent Psychology Studies Flawed?

The Dream

From our advertisers