What happens next… Peer Review

With the first Galaxy Zoo paper submitted (kudos to Kate and Anze!), we’d like to describe to you what happens next. What’s scientific publishing all about? How does it work? If you’ve followed the blog and the forum, you have a pretty good idea of the first part of the scientific process: discovery!

We set out on the Galaxy Zoo project in part to test whether spiral galaxies in different parts of the sky somehow have spins that align, as has been claimed by earlier work. Kate and Anze have commented on the motivation for this work and blogged about how we did find an effect, were startled by it and so started the bias test to understand it. Kate and Anze used the bias test data to show conclusively that in the case of Galaxy Zoo it was an effect with the observers and that the universe isn’t mad.

This is one of the amazing and unique things about science. Good scientists spend most of their time arguing against the effects they see in their own data, to avoid falling into traps of seeing only what they expect to see. To see how unique and amazing this is, try to imagine a politician arguing against a piece of legislation s/he is sponsoring! This process of double, triple, and quadruple-checking one’s own work is a very important part of science.

Once we were convinced that we really understood what is going on, we could then write up our conclusions in the form of a scientific paper. Steven wrote here about the process of writing a paper; Kate went through the same process Steven described. Over the past few weeks, she passed her paper around to the rest of the Galaxy Zoo team for comments. Kate’s paper has thus passed through the first check — her own examination of her results — and the second — amongst the team itself.

The next step in scientific research is to submit the paper to a journal. This has now happened, and the paper Land et al. (2008) (where “et al.” means “and the rest,” including YOU!!) has been submitted to the top UK journal Monthly Notices of the Royal Astronomical Society (MNRAS).

The editor of this journal will now select an anonymous referee who can comment on the scientific and technical merits of the paper. The referee is another astronomer or cosmologist whom the editor can ask for an expert assessment of the work. He or she will have a few weeks to read it, think about it, and then make a number of recommendations to the editor of the journal. There are three options. The referee can reject the paper outright. This generally happens very rarely, except in highly competitive top journals like Nature and Science. They can support publication of the paper, asking for only a few minor modifications. This also happens quite rarely, though! The most common outcome is for her to write a “referee report,” suggesting a number of modifications and ask for clarifications. The referee might have questions about some part of the analysis, suggest some alternative thoughts and ideas, or criticise the methodology. Sometimes referees can be hostile to a paper; but often, they are genuinely helpful and constructive.

After receiving the report, we get a few weeks to digest it and modify the paper according to the referee’s comments, and argue against the points raised that we disagree with. This process may repeat itself a number of times if the referee isn’t happy with our modifications, and so it can often take weeks and months for a paper to get to a decision by the editor (acceptance or rejection). If a referee is being particularly unreasonable, we can write to the editor requesting a new referee. In extreme circumstances, we could even choose to submit the paper to a different journal and hope for a more reasonable referee.

The whole process is generally known as peer review since the referee is a peer — a fellow scientist and expert in the field. If the paper is accepted, it will appear both in the online and print version of the journal after another few weeks or months. A paper accepted in such a journal is then considered peer-reviewed.

So, if Kate’s paper hasn’t yet been peer-reviewed why is the paper already “public”? It’s general practice in astrophysics to post papers as preprints on a web server called astro-ph. Astro-ph is updated daily to make all papers publicly accessible for anyone. Most people post their papers there when they submit them to journals so they are available immediately. Some wait till the paper is accepted. Thus, not everything on astro-ph is peer-reviewed! In fact, in cosmology, some like to submit preprints to astro-ph before submitting so to allow the community to comment before the draft is submitted to a journal.

It’s important to note that something said in a “peer-reviewed” paper isn’t necessarily true. The point of peer-review is to weed out obviously flawed paper whose logic has holes or whose data don’t support the conclusion. Knowing that a paper has been peer-reviewed should give you extra confidence that its results are believable – that means that an expert in the field has read through the paper and thinks its conclusions are believable.It’s really just the first step of proper “peer-review,” because the process continues. As the community of astrophysicists digests the paper, they too pass judgement on whetherthey consider the paper important and whether they believe the conclusion. Thus, in the years after publication, other astrophysicists might deem Land et al. (2008) a key paper and cite it in the future, commenting on it positively. Or they might disagree with it, but that would still be a sign that it was important enough to comment on. Or it might just fade into obscurity if astronomers don’t consider it important. That’s the historical legacy of a paper – and that’s the ultimate peer-review.