Radio Galaxy Zoo: “gold standard” images and improving our calibration

First off, the science team for Radio Galaxy Zoo wants to thank our volunteers for their continued clicks, discussion on Talk, and continued participation in the project. As of today, we have 892,582 classifications on RGZ and over 45,000 subjects completed from the FIRST-SDSS sample. We absolutely could not do this without you, and we’re working hard on turning the data into interesting science.

We want to let you know of some particular galaxies that will be appearing slightly more often in the interface. One of the things I’ve been working on for Radio Galaxy Zoo over the last month is finding better and smarter ways of combining clicks from independent classifiers into the “final answer” for each galaxy. For past Zooniverse projects, we’ve been able to do this using relatively simple methods – users are weighted a little bit by their consistency with other volunteers, but the final data product is mostly just the vote fractions for various tasks. However, the task in RGZ is a bit more complex, and the simpler methods of combining classifications are proving very difficult. In order to accurately combine the information each volunteer gives us, we need to establish a bit more common ground than we currently have.

To calibrate the clicks across all citizen scientists, we need to look at galaxies that the same people have done. The science team has started this by labeling the correct morphologies (to the best of our abilities) for a smaller, “gold standard” sample. We use these as seed weights in our data reduction – that lets us calibrate users who have done the gold standard galaxies. These results are propagated outward to the full sample by looking at other galaxies done by both calibrated and uncalibrated users, and so on. Kind of like pulling ourselves up by the bootstraps. 🙂

Result of the science team classification of a small sample of 100 RGZ galaxies. The height of the bar represents how well a particular science team member agreed with the others. As a group, the results show very good consistency overall, near 90%. Using the results from this sample, we can apply similar calibrations to the tens of thousands of galaxies that RGZ citizen scientists are helping us with.

What we’re missing right now, though, are galaxies that lots of citizen scientists have jointly classified. Since each galaxy is retired after 20 people classify it, the chances of seeing a particular galaxy is pretty low. Some members of the science team, including myself, recently sat down with a sample of 100 galaxies taken from a combination of random selection and ones you’ve identified on Talk as having interesting or non-trivial morphologies (bent jets, triple systems, giants, no IR counterparts, etc). These is what we’d like to use for calibration. However, only about a dozen users so far have done enough of this sample to give us enough data for calibration.

So, in order to help the accuracy of the data pipeline, we’ve chosen 20 “gold standard” galaxies that will eventually be shown to all volunteers. They won’t all be in one bunch (you should see one every five subjects or so) and you shouldn’t see any galaxies that you’ve classified before. We’ll label the galaxies on Talk – look for the hashtag #goldstandard. I hope that another positive outcome will be users getting to discuss interesting features in galaxies that they haven’t come across before. After you’ve done all 20 galaxies in the sample, your future classifications will be randomly selected as usual.

ARG0001e8e – a very nice core + hotspot system, and slightly challenging morphology to classify. This is one example of the 20 “gold standard” galaxies we’re trying to have everyone in RGZ classify.

Please let us know on Talk if you have any questions about this, and I’ll be happy to discuss it further. Thanks again for all of your help – we hope this will let us produce a more accurate RGZ product and science papers in the coming year!