Some thoughts on the UCL “Is Science Broken” debate

[Disclaimer: This was written the day after the event, and I took no notes. If I have misrepresented the opinions of any of the panel members then it is a result of my poor attention during, and poor memory following, the debate. My apologies to those involved if this is the case.]

For those that weren't aware, UCL hosted a talk and debate last night entitled “Is Science Broken? If so, how can we fix it?” Chris Chambers (@chriscd77) gave a talk about the recent introduction of Registered Reports in the journal Cortex. This was followed by a broader panel discussion on the problems facing science (and psychology in particular) and how initiatives, such as pre-registration, might be able to improve things. Alongside Chris, Dorothy Bishop (@deevybee), Sam Schwarzkopf (@sampendu), Neuroskeptic (@Neuro_Skeptic) and Sophie Scott (@sophiescott) took part in the debate, and David Shanks chaired.

First, I found Chris’ talk very informative and measured. Words such as “evangelist” are often bandied about on social media. Personally, I found him to be passionate about pre-registration but very realistic and honest about how pre-registration fits into the broader movement of “improving science”. He spend at least half of his talk answering questions that he has received following similar presentations over the last few months. I would guess about 90% of these questions were essentially logistical – “will I be able to submit elsewhere once I've collected the results?”, “couldn't a reviewer scoop my idea and publish whilst I’m data collecting?” It is obviously incumbent upon Chris, given he has introduced a new journal format, to answer these legitimate logistical questions clearly. I think he did a great job in this regard. I can’t help feeling some of these questions come from individuals who are actually ideologically opposed to the idea, trying to bring about death by a thousand cuts. Often these questions implicitly compare pre-registration to an “ideal” scenario, rather than to the current status quo. As a result, I feel Chris has to point out that their concern applies equally to the current publishing model. I may just be misreading, but if people are ideologically opposed to pre-registration I’d rather they just come out and say it instead of raising a million and one small logistical concerns.

On to the debate. This worked really well. It is rare to get five well-informed individuals on the same stage talking openly about science. There was a lot of common ground. First, everyone agreed there should be more sharing of data between labs (though the specifics of this weren't discussed in detail, so there may have been disagreement on how to go about doing this). Dorothy also raised legitimate ethical concerns about how to anonymise patient data to allow for data sharing. There was also common ground in relation to replication, though Chris and Neuroskeptic both cautioned against only replicating within-lab, and pushed for more between-lab replication efforts, relative to Sophie.

Where I think there was disagreement was in relation to the structures that we put in place to encourage good practice (or discourage bad practice). On several occasions Chris asked how we were going to ensure scientists do what they should be doing (replicating, data sharing, not p-hacking etc.). Essentially it boils down to how much scope we give individual scientists to do what they want to do. Pre-registration binds scientists (once the initial review process has been accepted) to perform an experiment in a very specific way and to perform specific statistics on the data collected. This should (though we need data, as pointed out by both Chris and Sam) decrease the prevalence of certain issues, such as p-hacking or the file drawer problem. You can’t get away from the fact that it is a way of controlling scientists though. I think some people find that uncomfortable, and to a certain extent I can understand why.  However, what is key to pre-registration is that it is the scientists themselves who are binding their own hands. It is masochistic rather than sadistic. Chris isn't telling any individual scientist how to run their experiment, he is simply asking scientists to clearly state what they are going to do before they do it. Given the huge multivariate datasets we collect in cognitive neuroscience, giving individuals a little less wiggle room is probably a good thing.

Sophie pointed out at the beginning of the debate that science isn't measured in individual papers. In one hundred years no-one will remember what we did, let alone that specific paper we published in Nature or Neuron (or Cortex). This is a reasonable point, but I couldn't quite see how it undermined the introduction of formats such as pre-registration. I don’t think anyone would claim a pre-registered paper is “truth”. The success (or failure) of pre-registration will be measured across hundreds of papers. The “unit” of science doesn't change as a result of pre-registration.

Where I found common ground with Sophie was in her emphasis on individual people rather than structures (e.g., specific journal formats). Certainly, we need to get the correct structures in place to ensure we are producing reliable replicable results. However, whilst discussing these structural changes we should never lose sight of the fact that science progresses because of the amazingly talented, enthusiastic, nerdy, focussed, well-intentioned, honest, funny, weird, clever people who design the experiments, collect the data, run the statistics and write the papers. The debate wonderfully underlined this point. We had five individuals (and a great audience) all arguing passionately about science. It is that raw enthusiasm that gives me hope about the future of science more than any change in journal format.

The academic parent #4 - things wot I learned

A brief list of things I have learnt in the space of less than a week:

1. Looking after your child on a daily basis is tiring. One day, it's a doddle. However, several consecutive days on your own and it starts to get to you. Both my wife and I have been shattered this last week. She has been adjusting to life back at work and I have been adjusting to the life of a full time carer. Hopefully we'll have a bit more energy this week and not have to go to bed at 9pm every night.
2. You're constantly thinking several hours ahead. When should I feed next, is it time for a nap, if I go for a walk now will she fall asleep and therefore not nap at home? Get your timings right and everything works wonderfully. Get them wrong and you end up spending 6 hours straight with a grumpy baby and no backup to help you out.
3. Trips to the shops are now the best thing ever. You get out the house and your child gets some fresh air and visual stimulation. Even if it's just walking down the road to buy some milk it can break up an afternoon into two more manageable chunks of time.
4. Don't sit down and have a cup of tea at the start of a nap. Before I knew it about an hour had gone by and I had been staring at the ceiling. I try to get straight on the computer so I can at least catch up on a few emails (whilst drinking my tea). That way I feel I haven't completely wasted my free time.
5. It's great. Try it.

The academic parent #3 – the return of paternity leave

Today I am back on paternity leave. My daughter is nine months old and my wife has returned to work. Rather than send our daughter straight to nursery, we thought it might be a good idea if I look after her full time for a month. We have done this for several reasons, which I will explain below. I had toyed with the idea of taking time off for a while (in addition to the paternity leave I took immediately after she was born), but it was a chance meeting at a conference dinner that really got me thinking. I was chatting to a lecturer who had returned to work following maternity leave, and she was explaining how her husband was taking 3 months of paternity leave. I explained I would like to do something similar, but given I’m currently a postdoc who is thinking about future jobs and perhaps needs a few more publications under his belt it wasn't the best timing for me. This was obviously crap, and luckily she called me out on it. There is never a “convenient” time to take paternity/maternity leave. There will always be an excuse. Since then the plan has been relatively set in stone, at least in my mind.

So, why take leave now? First, because I want to. I want to experience at least a small part of what my wife has been through the past nine months. It won’t be the same, it will be easier for me, but it will be close enough. More importantly, I want to spend more time with my daughter when she is still a baby. Second, because I can. I have an understanding boss and work in a supportive department. I may not be in this situation again so I should take these opportunities when they present themselves. Third, because it makes financial sense. Rather than take official paternity leave, I’m essentially taking all my annual leave in one go. I don’t think I have ever used up my allotted annual leave in a year. Most academics probably don’t. For this year at least that is what I am doing. This means, after a few months of statutory pay, my wife and I will both be on full pay for a month without nursery costs. This will help a lot. It would not have been financially viable for me to take official paternity leave, and therefore not be paid during this time, as we would not have had enough income between the two of us to cope. Fourth, because going back to work after nine months of maternity leave is a big deal, and having to cope with the emotion of leaving your child at the nursery seems like quite a lot to handle all at once. This way my wife can be at work knowing I’m looking after our daughter, at least for a few weeks.

Finally, because more men need to take time off work and look after their kids. I've had very mixed reactions to taking (only) one month off. Some have sounded shocked, others have asked what I’m going to do with my time (answer: look after my daughter). It’s that mix of reactions that made me realise it was the right thing to do on top of all the personal reasons listed above. If it’s still a shock that a man might take time off work to look after their child (and I reiterate, it is only one month so it really isn't that significant) then we live in a pretty weird society.

The academic parent #2 – work-life balance

A recent article in Nature nicely highlighted some of the difficulties associated with juggling both work and parenting responsibilities whilst trying to maintain some semblance of a social life. Needless to say, it isn’t easy. Whilst I found the article to be an honest and frank assessment of the trials and tribulations of parenthood and academia, I couldn’t help feeling that part of the discussion was missing.

We are introduced to several research active scientists who plan weeks ahead, call on friends/colleagues/parents to help with child care, work into the evening once their child has gone to bed, all in the quest to maintain their pre-child levels of work. For instance, during maternity/paternity leave one couple “planned to use [their child’s] nap times and evenings at home to work on data analysis, manuscripts and grant proposals”. Another example tells of how the “couple typically works side-by-side in their home office for three to four hours” after they have put their daughter to bed.

It’s great that these individuals are managing to find time to be a parent and be productive at the same time – although their social life seems to have suffered somewhat. However, my issue with the whole article is it presents two options (1) maintain previous work patterns and be a bad parent or (2) change work patterns but maintain the same working hours and be a good parent. At no point is the concept of working fewer hours brought in for consideration. I’m not saying this is the correct solution, but surely it is a viable option? Everyone agrees that academics can work long hours, reviewing papers in the evening, spending all night finishing a grant application, collecting data on the weekend. Given this common agreement, why is it not remotely conceivable that one might want to cut down on these hours once one has a child?

My issue with a lot of articles on “work-life balance” is actually that exact phrase. The word “balance” seems too positive a term for what is essentially a decision about what to sacrifice in your life. The above examples from the piece in Nature have, although not explicitly stated, sacrificed their social life in order to maintain work hours whilst spending time with their children. That’s fine, but this sacrifice should be explicitly acknowledged. Personally, I work slightly fewer hours than I used to (trying to be more productive with those hours I am in the office) and go out with friends a lot less. I have also sacrificed any time to myself in the evening. Once my child is asleep I spend time with my wife, as we rarely get time alone during the day. I consider this a relatively “balanced” life considering how much upheaval a baby causes in one’s life, but I have had to sacrifice quite a bit to reach that balance.

The point I am trying to make is that a lot of talk about “balance” is directed towards cramming more stuff into the same number of hours. Instead I think we should talk more openly about what is and what isn’t important. What we can give up and what we need to maintain. Only then should we discuss how we can use the finite number of hours allotted to us to carry out the tasks that we have prioritised.

The academic parent #1 - guilt and multiple personalities

Preamble

I plan to blog occasionally about being an academic and a parent. I’m not entirely sure what form this is going to take. I’ll try to keep each blog short. Most blogs are unlikely to have a moral or a valuable lesson to be learned. They are more likely to be a reflection on something that has happened to me recently that I feel others might be interested in. I believe this first post fits this description.

Note – I have been a parent for a few months. This is a ridiculously short period of time. All opinions are likely to change on a daily basis dependent on how much sleep I get the previous night and whether my daughter is teething. This should therefore very much be viewed as a snapshot.

Guilt and multiple personalities

My daughter was born seven months ago. She’s now crawling around, making strange loud noises from her mouth (and other body parts), and smiling copiously. It’s a genuine joy spending time with her on a regular basis, and a genuine pain having to get up early every morning. It’s the best thing I've done in my life.

I've been meaning to blog about parenthood in some form for a while now (seven months in fact!), but strangely enough have had very little time to do it. The first few weeks were a complete blur. I took two weeks off, and then went back part time for a further 4 weeks. Altogether this added up to about a month of paternity leave. Luckily I work for an organisation that provides up to a month paternity leave at full pay. I count my blessings (I don't know how people cope with two weeks - I would have taken more if I could).

Returning to work was a strange sensation, and this is the topic of this first blog. When I returned to work it seemed like I had been away for a long time, but of course nothing had changed. Slotting back in to the regular daily routine was surprisingly easy – like sitting down in a comfortable arm chair. The thing I found most difficult to deal with, mostly due to how unexpectedly it appeared, was guilt. When at work, I was thinking about being at home and felt guilty about leaving my wife alone with the baby. When I was at home, I was thinking about being at work and felt guilty about not getting enough work done. I then started to feel guilty about thinking about work at home and thinking about home at work. It all got very confusing and meant I wasn't working very hard nor being that great a parent.

I’ll return to guilt shortly. First, I want to take a small detour to discuss multiple personalities. I've always been a bit suspicious of people who act completely differently dependent on context. Of course we all moderate our behaviour based on where we are and who we are talking to, but I've always found it jarring when someone I know well in a work situation acts like a different individual in a social context. Having a baby has changed this. I now do whatever it is I have to do to make my daughter smile. Baby smiles are like crack – highly addictive. If I have to sing a stupid song, pull a stupid face, blow raspberries, so be it. I will sacrifice my dignity at the altar of my child's happiness. I have never been a particularly frivolous silly individual, so I have had to learn how to behave in this way. I've essentially developed a new personality that would be wholly inappropriate in a work setting. I now find it surprisingly easy to switch between these modes of thought (if you can describe a set of behaviours that mostly consist of doing stupid things as a mode of thought).

How does this relate to the guilt I mentioned earlier? Put simply, having multiple personalities has cured my guilt. My work persona now doesn't think too much about home. Equally, my home persona cares not for the seriousness of work. The compartmentalisation has made me more efficient at work and a better parent at home.

My Negative CV

I present to you my negative CV. Rather than list all my successes (as I do here), here I’m listing all the jobs I haven’t been offered, all the papers I’ve had rejected, and all the awards I haven’t been given.

Before I start, I should probably note that I’m not doing badly at present. I had a successful PhD, and am in my third year of a five year post-doc. I have several publications, and was even lucky enough to win a prize for my PhD work. It would be dishonest to claim I’m not doing reasonably well, but I certainly know individuals with ‘stronger’ CVs – prestigious fellowships, publications in ‘big’ journals etc. My point in opening up my CV is more to show the extent of rejection that has gone with the successes I have had. This might offer hope to PhD students, suggesting that rejections don’t spell the end of their career, or it could provoke anxiety, wondering how they could put up with so much rejection (or even that they've been at the recieving end of a lot more rejection). Regardless, I hope that the information is useful for some. Whether or not potential future employees will regard it as ‘useful’ is another matter, but one I will have to cope with when the time comes.

Educations & Jobs:

2005

Apply for PhD position @ Cambridge – rejected

Apply for PhD position @ York – rejected

2009

Apply for post-doc position @ Cambridge – rejected

Apply for College fellowship @ Cambridge – rejected

2010

Apply for Wellcome fellowship – rejected

Apply for MRC fellowship – rejected

Apply for ESRC fellowship – rejected

Apply for post-doc position @ Oxford – rejected

Apply for post-doc position @ Birkbeck – rejected

2011

Apply for British Academy fellowship – rejected

Awards and Prizes

2010

Nominated for BPS postgrad award – nope

2011

Nominated for BPS postgrad award – nope

Nominated for EPS Frith Prize award – nope

2013

Nominated for BPS postgrad award – nope

Publications

2008

Submit to Neuroimage – rejected

2010

Submit to Neuropsychologia – rejected

2012

Submit to Nature Neuroscience – rejected

Submit to Neuron – rejected

Submit to Science – rejected

2013

Submit to PLOS: Biology – rejected

Submit to PNAS – rejected

Submit to PLOS: Biology – rejected

Submit to PNAS – rejected

As can been seen, I’ve been rejected a few times since 2005 (this doesn't include undergrad courses I was rejected from). I’ve no idea what the average rate of rejection is for someone at a similar stage in their career. I was relatively selective in applying for post-docs in or around the London area (for personal reasons), so perhaps applied for fewer jobs than others might do when finishing their PhD. Who knows. What is clear is that in order to have even a small amount of success you need to keep banging on the door until someone lets you in. The only reason I got into Cambridge to do a PhD was because I went away, did an MSc, and reapplied the following year with a stronger CV. The only reason I won an award for my PhD work was because my PhD supervisor was willing to repeatedly nominate me over three successive years.

I think two points stem from this: (1) get used to rejection, it is part of the job and (2) find a way to channel rejection into something productive. The easiest thing to do when you have a paper rejected is to sit on it for a few months – the best thing to do is to work on it straight away. That exasperated annoyance you get when reviewers/editors haven’t realised the brilliance of your manuscript? Use it to make your paper better and submit it somewhere else (although perhaps wait a day or two just to calm yourself down a little bit first…).

Note – although every attempt has been made to ensure this CV is accurate, it was actually surprisingly difficult to retrace my steps (at least is terms of rejection). It seems even electronic memory has a positive bias.

Replication and methods sections

Wow, things got a bit shouty there didn’t they. If you’re not up to speed, some people like replication, others don’t, and they don’t seem to get along very well. In an effort to remove all the interest out of this topic I thought it best to write a boring post about methods sections.

First things first, I like replication (who doesn’t!). I haven’t been involved in attempting to perform direct replications of other peoples work (at least not until very recently), but I replicate my own findings as much as possible to persuade myself of the validity of my results. I’m broadly supportive of the recent ‘replication movement’ although I’m not sure how novel an idea it really is given plenty of areas in psychology have been replicating for many a year.

Here is the small point I want to make, and it relates to the issue of being able to replicate purely on the basis of the methods section from a paper from another lab. The pro-replicators often state that if methods sections are written appropriately, anyone should be able to perform a replication of the study. However, this strong statement seems to me to be a bit naïve. I have two reasons for thinking this:

1.  A good methods section should include ‘all the necessary details in order to perform the experiment again’ however it should also exclude ‘any extraneous detail’. Without the latter then methods section could read something along the lines of:

“Procedure – each participant was welcomed in the lobby of the ground floor of 17 Queen Square, London, UK. The experimenter shook their right hand and encouraged them to enter the lift. The experimenter pressed the button for the 2^nd floor. Doors closed within 10s of this button press…….The participant was asked to take a seat in front of the computer screen with both feet firmly on the floor such that their upper and lower legs formed a right angle with each other.

You’re probably reading this thinking this is absurd, and I agree. My (small) point is that it can sometimes be difficult to decide what is ‘necessary detail’ and what is ‘extraneous’. We make a judgement called based on our experience and knowledge of the literature. This will always be the case.  The issue being, one person’s idea of ‘extraneous’ will sometimes be different from another person’s – I wouldn’t dream of stating that the participant held a hot beverage before starting the experiment, but some think this is obviously relevant in certain situations.

2.  A well written methods section doesn’t allow ‘anyone’ to replicate the experiment. If I gave a well written methods section from a psychology journal to a historian of art I wouldn’t expect them to be able to replicate the experiment (or, if they managed, they wouldn’t do it very well). This argument applies to a lesser extent within sub-divisions of psychology as well – I would expect a cognitive psychologist who studies memory to be able to replicate my experiments more thoroughly than a social psychologist (and vice versa). If this wasn’t the case, why do we think it is useful giving students experience in running experiments if not to ‘learn the technique’?

This isn’t to say that we can’t and shouldn’t be able to replicate based purely on the methods section of someone’s paper. It’s simply to say, writing a methods section is hard and requires some amount of subjectivity with regards to what to include. Also, replication is hard – it’s impossible to replicate exactly – judgement calls have to be made about whether subtle differences between the original and the replication attempt matter.

Given all this, I heartily recommend talking to each other more (preferably in a civil manner). If I wanted to replicate someone’s experiment I would email them and ask them as many questions as possible. They don’t have a right to be involved, or contribute, or have a say in the experiment I am running, but it would be foolish for me to not want to communicate with them. Equally, I would be honoured if someone thought my experiment was worthy of replication. I would of course be nervous – what if it doesn’t replicate!? – and worried – did I make a mistake previously!? – but all these things are natural consequences of being a human being with a vested interest in my own research. I hope I’d be grown up enough to deal with those anxieties. Time will tell…