Some thoughts on the UCL “Is Science Broken” debate

[Disclaimer: This was written the day after the event, and I took no notes. If I have misrepresented the opinions of any of the panel members then it is a result of my poor attention during, and poor memory following, the debate. My apologies to those involved if this is the case.]

For those that weren't aware, UCL hosted a talk and debate last night entitled “Is Science Broken? If so, how can we fix it?” Chris Chambers (@chriscd77) gave a talk about the recent introduction of Registered Reports in the journal Cortex. This was followed by a broader panel discussion on the problems facing science (and psychology in particular) and how initiatives, such as pre-registration, might be able to improve things. Alongside Chris, Dorothy Bishop (@deevybee), Sam Schwarzkopf (@sampendu), Neuroskeptic (@Neuro_Skeptic) and Sophie Scott (@sophiescott) took part in the debate, and David Shanks chaired.

First, I found Chris’ talk very informative and measured. Words such as “evangelist” are often bandied about on social media. Personally, I found him to be passionate about pre-registration but very realistic and honest about how pre-registration fits into the broader movement of “improving science”. He spend at least half of his talk answering questions that he has received following similar presentations over the last few months. I would guess about 90% of these questions were essentially logistical – “will I be able to submit elsewhere once I've collected the results?”, “couldn't a reviewer scoop my idea and publish whilst I’m data collecting?” It is obviously incumbent upon Chris, given he has introduced a new journal format, to answer these legitimate logistical questions clearly. I think he did a great job in this regard. I can’t help feeling some of these questions come from individuals who are actually ideologically opposed to the idea, trying to bring about death by a thousand cuts. Often these questions implicitly compare pre-registration to an “ideal” scenario, rather than to the current status quo. As a result, I feel Chris has to point out that their concern applies equally to the current publishing model. I may just be misreading, but if people are ideologically opposed to pre-registration I’d rather they just come out and say it instead of raising a million and one small logistical concerns.

On to the debate. This worked really well. It is rare to get five well-informed individuals on the same stage talking openly about science. There was a lot of common ground. First, everyone agreed there should be more sharing of data between labs (though the specifics of this weren't discussed in detail, so there may have been disagreement on how to go about doing this). Dorothy also raised legitimate ethical concerns about how to anonymise patient data to allow for data sharing. There was also common ground in relation to replication, though Chris and Neuroskeptic both cautioned against only replicating within-lab, and pushed for more between-lab replication efforts, relative to Sophie.

Where I think there was disagreement was in relation to the structures that we put in place to encourage good practice (or discourage bad practice). On several occasions Chris asked how we were going to ensure scientists do what they should be doing (replicating, data sharing, not p-hacking etc.). Essentially it boils down to how much scope we give individual scientists to do what they want to do. Pre-registration binds scientists (once the initial review process has been accepted) to perform an experiment in a very specific way and to perform specific statistics on the data collected. This should (though we need data, as pointed out by both Chris and Sam) decrease the prevalence of certain issues, such as p-hacking or the file drawer problem. You can’t get away from the fact that it is a way of controlling scientists though. I think some people find that uncomfortable, and to a certain extent I can understand why.  However, what is key to pre-registration is that it is the scientists themselves who are binding their own hands. It is masochistic rather than sadistic. Chris isn't telling any individual scientist how to run their experiment, he is simply asking scientists to clearly state what they are going to do before they do it. Given the huge multivariate datasets we collect in cognitive neuroscience, giving individuals a little less wiggle room is probably a good thing.

Sophie pointed out at the beginning of the debate that science isn't measured in individual papers. In one hundred years no-one will remember what we did, let alone that specific paper we published in Nature or Neuron (or Cortex). This is a reasonable point, but I couldn't quite see how it undermined the introduction of formats such as pre-registration. I don’t think anyone would claim a pre-registered paper is “truth”. The success (or failure) of pre-registration will be measured across hundreds of papers. The “unit” of science doesn't change as a result of pre-registration.

Where I found common ground with Sophie was in her emphasis on individual people rather than structures (e.g., specific journal formats). Certainly, we need to get the correct structures in place to ensure we are producing reliable replicable results. However, whilst discussing these structural changes we should never lose sight of the fact that science progresses because of the amazingly talented, enthusiastic, nerdy, focussed, well-intentioned, honest, funny, weird, clever people who design the experiments, collect the data, run the statistics and write the papers. The debate wonderfully underlined this point. We had five individuals (and a great audience) all arguing passionately about science. It is that raw enthusiasm that gives me hope about the future of science more than any change in journal format.