Change Your Life: Living Skillfully

I link to scientific studies a lot here – for example, in my previous post (Health Research and News: Link Roundup). I’m aiming for an audience of intelligent, but not necessarily scientifically educated, readers, so I thought I would post briefly on one of the key things which the media, in particular, often miss when reporting scientific studies.

Two Scientists by Robin Hutton

At their most basic, scientific studies are investigations of how measurable things in the observable world interact. You identify two (or more, but usually two) things of interest – let’s call them Thing A and Thing B – that you think might interact and that are found in the same circumstances. You then find some way of measuring each thing. You gather as many examples as you can, given your resources, and you look at the variations in the two things and see if there is a clear link between them, based on a statistical treatment of the two sets of measurements.

An important distinction to make is between experiments and non-experiments. In an experiment, the scientists doing the study somehow manipulate one of the two things – let’s say Thing A for the sake of illustration – and measure the other, Thing B. If done right, this can show that Thing A is (or isn’t) the cause of Thing B. Once you know that, you can apply the science by coming up with ways to change Thing B by changing Thing A, which is very useful if Thing B is not something you know how to change directly. For example, if you show that doing particular exercises causes your muscles to grow, and someone wants bigger muscles, they now have a way of achieving that goal.

In a non-experimental study, though, the scientists are just observing and measuring both of the things, not manipulating either of them, and in this case, they cannot prove a causal connection between a change in Thing A and a change in Thing B. They can show an association, but there is no way of knowing – without doing an experiment, where they do manipulate one of the variables – whether Thing A is causing Thing B, Thing B is causing Thing A, or both of them are caused by a Thing C which you didn’t study.

The problem arises when the media seize on a non-experiment and report it as if it showed a cause-effect relationship. It doesn’t; it can’t. A cause-effect relationship may be plausible, even highly plausible, but it isn’t scientifically established unless you have done an experiment – that is, manipulated one of the things and seen it affect the other.

Let’s take some examples from yesterday’s post.

The first set of studies, into arts and cognition, were concerned in part with the question, “Are smart people drawn to the arts or does arts training make people smarter?” This is a question about cause and effect. A couple of the studies go some way towards answering the question, because they measure a manipulated variable – the level of training people have received – against their specific skills. Others of the studies just show an “association” between two variables. The report on the studies correctly makes the distinction.

Both of the aromatherapy studies (the one that showed no healing benefit of aromatherapy and the one that showed a gradual effect of aromatherapy on stress) had variables (exposure to aromas) that were manipulated by the researchers. They were experiments, and they show (or, in the first case, don’t show) a cause and effect relationship. Likewise with the music therapy studies. If they had just been studies in which people were asked how much music they listened to and then the other variable (immune status or depression) was also measured, it could have been the case, for example, that depressed people listen to less music because of their depression. You wouldn’t know.

The stress/immune system/cancer study is an interesting one and a classic illustration of my point. I’ve been guilty of believing a press release, which says:

A new study led by Prof. Shamgar Ben-Eliyahu, from Tel Aviv University’s Department of Psychology, has shown scientifically [my emphasis] that psychological and physiological stress prior to, during and after surgery has a biological impact that impairs immune system functioning. This impairment bears down on disease progression, he says, especially at the critical point during oncological surgery when a primary tumor is being removed.

The study was published in the journal Brain, Behaviour, and Immunity (2007).

The problem is that when I use my special powers as a student at Massey University to access the full text of all three articles with Prof. Ben-Eliyahu’s name on published in that journal in 2007, the only article that could reasonably be described as “new” (in the October edition) is not a report of a study but a survey of the literature. The other two, published in May, don’t discuss psychological stress at any length and they are not studies on its effects. So really, this is a false claim in a press release which I should have investigated more carefully.

The breakfast study is also a good one to look at. It does not (as I initially said – incorrectly) show that eating breakfast helps reduce obesity. I quote from the Eurekalert report:

Researchers examined the association [my emphasis] between breakfast frequency and five-year body weight change in more than 2,200 adolescents, and the results indicate that daily breakfast eaters consumed a healthier diet and were more physically active than breakfast skippers during adolescence. Five years later, the daily breakfast eaters also tended to gain less weight and have lower body mass index levels — an indicator of obesity risk — compared with those who had skipped breakfast as adolescents.

Mark Pereira, Ph.D., corresponding author on the study, points out that this study extends the literature on the topic of breakfast habits and obesity risk because of the size and duration of the study. “The dose-response findings between breakfast frequency and obesity risk, even after taking into account physical activity and other dietary factors, suggests that eating breakfast may have important effects on overall diet and obesity risk, but experimental studies are needed to confirm these observations [my emphasis],” he added.

The study shows an association, not cause and effect, and one of the authors correctly points out that experimental studies are needed to establish that cause and effect.

I’ll skip over the rest, as I think the point has been made, except that I will briefly discuss the last one, concerning the impact of the public smoking ban in France on hospital cardiac admissions. Combined with the results in Italy, Ireland and Scotland, I believe this counts as an experiment – an experiment on a national scale, where the variable – public smoking – was manipulated by the government rather than the researchers. As such, it can be argued that it shows a clear cause and effect relationship between public smoking and the incidence of heart disease (for non-smokers as well as smokers, by the way).

No related posts.