Saturday, 31 May 2014

Replication and methods sections

Wow, things got a bit shouty there didn’t they. If you’re not up to speed, some people like replication, others don’t, and they don’t seem to get along very well. In an effort to remove all the interest out of this topic I thought it best to write a boring post about methods sections.

First things first, I like replication (who doesn’t!). I haven’t been involved in attempting to perform direct replications of other peoples work (at least not until very recently), but I replicate my own findings as much as possible to persuade myself of the validity of my results. I’m broadly supportive of the recent ‘replication movement’ although I’m not sure how novel an idea it really is given plenty of areas in psychology have been replicating for many a year.

Here is the small point I want to make, and it relates to the issue of being able to replicate purely on the basis of the methods section from a paper from another lab. The pro-replicators often state that if methods sections are written appropriately, anyone should be able to perform a replication of the study. However, this strong statement seems to me to be a bit naïve. I have two reasons for thinking this:

1.  A good methods section should include ‘all the necessary details in order to perform the experiment again’ however it should also exclude ‘any extraneous detail’. Without the latter then methods section could read something along the lines of:

“Procedure – each participant was welcomed in the lobby of the ground floor of 17 Queen Square, London, UK. The experimenter shook their right hand and encouraged them to enter the lift. The experimenter pressed the button for the 2nd floor. Doors closed within 10s of this button press…….The participant was asked to take a seat in front of the computer screen with both feet firmly on the floor such that their upper and lower legs formed a right angle with each other.

You’re probably reading this thinking this is absurd, and I agree. My (small) point is that it can sometimes be difficult to decide what is ‘necessary detail’ and what is ‘extraneous’. We make a judgement called based on our experience and knowledge of the literature. This will always be the case.  The issue being, one person’s idea of ‘extraneous’ will sometimes be different from another person’s – I wouldn’t dream of stating that the participant held a hot beverage before starting the experiment, but some think this is obviously relevant in certain situations.

2.  A well written methods section doesn’t allow ‘anyone’ to replicate the experiment. If I gave a well written methods section from a psychology journal to a historian of art I wouldn’t expect them to be able to replicate the experiment (or, if they managed, they wouldn’t do it very well). This argument applies to a lesser extent within sub-divisions of psychology as well – I would expect a cognitive psychologist who studies memory to be able to replicate my experiments more thoroughly than a social psychologist (and vice versa). If this wasn’t the case, why do we think it is useful giving students experience in running experiments if not to ‘learn the technique’?

This isn’t to say that we can’t and shouldn’t be able to replicate based purely on the methods section of someone’s paper. It’s simply to say, writing a methods section is hard and requires some amount of subjectivity with regards to what to include. Also, replication is hard – it’s impossible to replicate exactly – judgement calls have to be made about whether subtle differences between the original and the replication attempt matter.

Given all this, I heartily recommend talking to each other more (preferably in a civil manner). If I wanted to replicate someone’s experiment I would email them and ask them as many questions as possible. They don’t have a right to be involved, or contribute, or have a say in the experiment I am running, but it would be foolish for me to not want to communicate with them. Equally, I would be honoured if someone thought my experiment was worthy of replication. I would of course be nervous – what if it doesn’t replicate!? – and worried – did I make a mistake previously!? – but all these things are natural consequences of being a human being with a vested interest in my own research. I hope I’d be grown up enough to deal with those anxieties. Time will tell…