Wow, things got a bit shouty there didn’t they. If you’re not up to speed, some people like replication, others don’t, and they don’t seem to get along very well. In an effort to remove all the interest out of this topic I thought it best to write a boring post about methods sections.
First things first, I like
replication (who doesn’t!). I haven’t been involved in attempting to perform direct
replications of other peoples work (at least not until very recently), but I replicate
my own findings as much as possible to persuade myself of the validity of my
results. I’m broadly supportive of the recent ‘replication movement’ although
I’m not sure how novel an idea it really is given plenty of areas in psychology
have been replicating for many a year.
Here is the small point I want to
make, and it relates to the issue of being able to replicate purely on the
basis of the methods section from a paper from another lab. The pro-replicators
often state that if methods sections are written appropriately, anyone should
be able to perform a replication of the study. However, this strong statement
seems to me to be a bit naïve. I have two reasons for thinking this:
1. A good methods section should
include ‘all the necessary details in order to perform the experiment again’
however it should also exclude ‘any extraneous detail’. Without the latter then
methods section could read something along the lines of:
“Procedure – each participant was welcomed in the lobby of the ground
floor of 17 Queen Square, London, UK. The experimenter shook their right hand
and encouraged them to enter the lift. The experimenter pressed the button for
the 2nd floor. Doors closed within 10s of this button press…….The
participant was asked to take a seat in front of the computer screen with both
feet firmly on the floor such that their upper and lower legs formed a right angle
with each other.
You’re probably reading this
thinking this is absurd, and I agree. My (small) point is that it can sometimes
be difficult to decide what is ‘necessary detail’ and what is ‘extraneous’. We
make a judgement called based on our experience and knowledge of the
literature. This will always be the case. The issue being, one person’s idea of ‘extraneous’
will sometimes be different from another person’s – I wouldn’t dream of stating
that the participant held a hot beverage before starting the experiment, but
some think this is obviously relevant in certain situations.
2. A well written methods section
doesn’t allow ‘anyone’ to replicate the experiment. If I gave a well written
methods section from a psychology journal to a historian of art I wouldn’t
expect them to be able to replicate the experiment (or, if they managed, they
wouldn’t do it very well). This argument applies to a lesser extent within
sub-divisions of psychology as well – I would expect a cognitive psychologist
who studies memory to be able to replicate my experiments more thoroughly than
a social psychologist (and vice versa). If this wasn’t the case, why do we think it is useful
giving students experience in running experiments if not to ‘learn the
technique’?
This isn’t to say that we can’t
and shouldn’t be able to replicate based purely on the methods section of
someone’s paper. It’s simply to say, writing a methods section is hard and requires
some amount of subjectivity with regards to what to include. Also, replication
is hard – it’s impossible to replicate exactly – judgement calls have to be
made about whether subtle differences between the original and the replication attempt
matter.
Given all this, I heartily
recommend talking to each other more (preferably in a civil manner). If I wanted
to replicate someone’s experiment I would email them and ask them as many
questions as possible. They don’t have a right to be involved, or contribute,
or have a say in the experiment I am running, but it would be foolish for me to
not want to communicate with them. Equally, I would be honoured if someone
thought my experiment was worthy of replication. I would of course be nervous –
what if it doesn’t replicate!? – and worried – did I make a mistake previously!?
– but all these things are natural consequences of being a human being with a
vested interest in my own research. I hope I’d be grown up enough to deal
with those anxieties. Time will tell…
