“Can’t Believe It” (we deny research findings that defy our beliefs)

So, I have been running a little experiment on twitter. Oh well, it doesn’t really deserve the term “experiment” – at least in an academic vocabulary – because there certainly are no treatment effects or control groups. It does deserve the term “little” though, because there are only four observations.

My experiment was to post a few recent findings from academic research that some might find mildly controversial or – as it turns out – offending. These four hair raising findings were 1) selling junk food in schools does not lead to increased obesity, 2) family-friendly workplace practices do not improve firm performance (although they do not decrease them either), 3) girls take longer to heal from concussions, 4) firms headed up by CEOs with broader faces show higher profitability.

Only mildly controversial I’d say, and only to some. I was just curious to see what reactions it would trigger. Because I have noticed in the past that people seem inclined to dismiss academic evidence if they don’t like the results. If the results are in line with their own beliefs and preconceptions, its methods and validity are much less likely to be called stupid.

Selling junk food in schools does not lead to increased obesity is the finding of a very careful study by professors Jennifer Van Hook and Claire Altman. It provides strong evidence that selling junk food in schools does not lead to more fat kids. One can then speculate why this is – and their explanation that children’s food patterns and dietary preferences get established well before adolescence may be a plausible one – but you can’t deny their facts. Yet, it did lead to “clever” reactions such as “says more about academic research than junk food, I fear…”, by people who clearly hadn’t actually read the study.

Family-friendly workplace practices do not improve firm performance is another finding that is not welcomed by all. This large and competent study, by professors Nick Bloom, Toby Kretschmer and John van Reenen, was actually read by some, be it clearly without a proper understanding of its methodology (which, indeed, it being an academic paper, is hard to fully appreciate without proper research methodology training). It led to reactions that the study was “in fact, wrong”, made “no sense”, or even that it really showed the opposite; these silly professors just didn’t realise it.

Girls take longer to heal from concussions is the empirical fact established by Professor Tracey Covassin and colleagues. Of course there is no denying that girls and boys are physiologically different (one cursory look at my sister in the bathtub already taught me that at an early age), but the aforementioned finding still led to swift denials such as “speculation”!

That firms headed up by CEOs with broader faces achieve higher profitability – a careful (and, in my view, quite intriguing) empirical find by my colleague Margaret Ormiston and colleagues – triggered reactions such as “sometimes a study tells you more about the interests of the researcher, than about the object of the study” and “total nonsense”.

So I have to conclude from my little (academically invalid) mini-experiment that some people are inclined to dismiss results from research if they do not like them – and even without reading the research or without the skills to properly understand it. In contrast, other, nicer findings that I had posted in the past, which people did want to believe, never led to outcries of bad methodology and mentally retarded academics and, in fact, were often eagerly retweeted.

We all look for confirmation of our pre-existing beliefs and don’t like it much if these comfortable convictions are challenged. I have little doubt that this also heavily influences the type of research that companies conduct, condone, publish and pay attention to. Even if the findings are nicer than we preconceived (e.g. the availability of junk food does not make kids consume more of it), we prefer to stick to our old beliefs. And I guess that’s simply human; people’s convictions don’t change easily.

9 Comments on ““Can’t Believe It” (we deny research findings that defy our beliefs)”

  1. Steve Phelan says:

    Good old confirmation bias at work.

  2. Peter Klein says:

    I love controversial research, and therefore I love this blog post.

  3. Pat Hastings says:

    But then, when said people see research that conforms to pre-existing beliefs, we hear, “Oh, that is so obvious!”

    (see http://www.amazon.com/Everything-Is-Obvious-Once-Answer/dp/0385531680).

  4. srp says:

    The advantage of a background in debate or litigation is that you can train yourself to ask, what I had to defend this study (whose finding I don’t like)? or what if I had to attack this study (whose findings I do like?). I don’t know what it says that I find the latter mental operation more natural.

  5. Funny how confirmation bias works– both for the audience being critiqued and for the writer critiquing it.

    One issue embedded within confirmation bias is that we reject the information without considering it critically, and closely. Take for example the critique of Family Friendly Work Life Policies you link to at HuffPo — that critique drills down to the level of construct definition (e.g., what is a FFWL policy) and challenges the composition of the empirical variables.

    That article also challenges the idea that FFWP can be separated from “good management practices”. Can ‘wet’ be separated from ‘water’?

    Third, the article challenges the authors of the study for not including previous empirical work in their lit review and in their construct definition, and points out that the study’s construal and design is largely detached from the WorkLife research stream (and not with an alternative voice, but with a deaf ear).

    I like to counter my biases with a well-done, nearly bullet-proof research study. The WorkLife study you cite is not one of those.

    A last issue re: confirmation bias, and scientific progress in general: one study with a counter-conventional finding should open the conversation, but cannot prove or disprove until it is supported by just a little bit more empirical work. This is especially true when there is such a body of well-done empirics to support the ‘conventional’ wisdom.

    • Economist says:

      I think this comment proves this article perfectly. What is poor science in one discipline (not doing a detailed construct-based definition and harmonizing with prior FFWP research) is good science in another (using objective measures without prior prejudice)

  6. JuniorProf says:

    It’s one level of troubling when people not trained in research methods dismiss a study because its findings are inconsistent with their priors. It’s a whole other level of troubling when actual scholars do the same thing

  7. […] earlier post – “can’t believe it” – triggered some bipolar comments (and further denials); also to what extent this behaviour […]

  8. ProfDC says:

    I’ll see it when I believe it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s