Our New Infrastructure

As some of you may already know I’m relatively new to the Galaxy Zoo team having joined in January.  One of the main reasons I was brought on board the team was to help plan and implement a more scalable web architecture for the future.

Galaxy Zoo 2 launched on Tuesday and was our first live test of our new approach. I’m pleased to say that apart from a small hiccup with our database server on day two we’ve managed to collect 3.7 million classifications in just over three days – well done everyone!

*Warning here begins gory technical details!* You can safely stop reading here unless you are a techy.

Galaxy Zoo is now hosted on Amazon Web Services.  We are using a combination of S3 (their storage solution) for hosting the galaxy images, EC2 (their compute service) for the Galaxy Zoo website (and our super-secret API) and EBS (Elastic Block Store) for instant backups of our databases. The great thing about Amazon Web Services is that you pay for what you use (by the hour) and more significantly you can scale up to as many servers as you like to handle the load you are experiencing when busy.

Those of you who are particularly observant may have also noticed that the Zoo 2 site is now written in the web framework Ruby on Rails. Rails is great fit for us, it’s a modern web framework that follows a great design pattern (MVC) and encourages best practice development.

For those of you who have found the last two paragraphs interesting, I’m going to find a place where I can write about what we are up to on the technical side of things. Watch this space!

Cheers,

Arfon.

'My Galaxies' is Back!

This morning we’ve added back in the ‘My Galaxies’ feature to the Zoo 2 site. The ‘My Galaxies’ page increases significantly the load on our servers and so we had to disable it very soon after launch due to the sheer number of people coming to the site. So that we can reintroduce ‘My Galaxies’ without risking slowing the site down too much, you will only be able to see the ‘My Galaxies’ page if you registered more than three days ago. This three day limit is a rolling time window, i.e. all Zooites from Zoo 1 will be able to access ‘My Galaxies’ now and new users will be able to access it three days after they registered.

At the moment ‘My Galaxies’ is only showing the 21 most recently ‘favourited’ galaxies. Don’t worry though, all of your favourites are being stored and soon you will be able to access them all. In addition you’ll soon be able to see your classification history and the classifications you made.

Cheers,

Arfon and the Galaxy Zoo Team.

This is my first time….

This is my first Galaxy Zoo blog posting and being one of the oldest members of the team (42!) I’m a bit lost with this new technology – sign of old age. Galaxy Zoo 2 was launched only a few days ago and Chris L rang me this morning to tell me we already have 2 million classification, so you guys are averaging a million galaxies a day.

That is staggering for us astronomers as we are usually expect our experiments to take a lot longer. For example, if one wants to use a telescope to study something in the sky, one must write a proposal 6 months in advance, submit it for scrutiny, and then await your allocation of time on a telescope. The process can take nearly a year and then after your night staring at the stars, it can take a further year to analyse the data (assuming it wasn’t cloudy!). Only then are we ready to ask questions of the data and test our observations against our original hypothesis written two years ago in a haste!

With the Zoo, it’s all a little too quick! For example, I can ask the question “how many galaxies have a bar through the middle of them” and typically I would embark on a career-long quest to answer this fundamental question. I may even recruit some poor graduate student to eyeball 50,000 galaxies to answer the question (like they did with Kevin!). But now, two days after the launch, we already have the data to address this question and it’s a little too fast for an old-timer like me. This story does however demonstrate the impact of technology on science. Thirty years ago the arrival of CCD digital detectors on telescope revolutionized the way we did astronomy. We could see deeper and faster than with photographic plates and surveys like the Sloan Digital Sky Survey (SDSS) became a reality. The internet is clearly the revolutionary technology of this generation of astronomer. Youngsters like Chris and Steven have embraced it and Galaxy Zoo is an amazing demonstration of how powerful this new tool can be used to address new questions.

The future does look bright as our ability to build bigger and better digital detectors allow us to scan the heavens faster and deeper. This year a new telescope called Pan-STARRS1 will start operations and will scan the northern hemisphere repeatedly looking for killer asteroids and supernovae. This new technological advance will open up the time-domain in astronomy and soon we maybe be showing you movies of each individual galaxy. How many galaxies change in time? Who knows… Therefore, science and technology are intimately linked. The desire of scientists to do better science drives technology, while new technologies open up new science capabilities. We need to fund both of these endeavours. I will end here with my thanks again for all your clicks and encourage you to keep going. You are part of a revolution and it’s a little scary, as all revolutions are.

This evening's downtime

Some of you may have noticed that we had some unscheduled downtime this evening. At about 18:40 our database server failed and due to the sheer number of classifications that you have all been making it took significantly longer than expected to restore the database from backup. I can assure you that we didn’t loose any of your classifications. Your hard work has not been wasted! I’m pleased to say that the site is now running well again and tomorrow I’m going to be working on making our database more resilient so that this doesn’t happen again. Happy classifying! The Galaxy Zoo Team

It's all happening (in the Zoo)

It’s been quite a day. I think – despite slowness at times – we’ve just about survived the onslaught so far. We’ll see what happens tomorrow – I don’t think we’re quite over the peak yet. The great thing about that is that the sheer levels of enthusiasm for the project, for the galaxies and for taking part in science should keep rising too. There will be more reflective posts, I promise, and lots more good stuff to come but I can’t resist linking to Sarah’s reworking of a Simon and Garfunkel classic.

It is, indeed, all happening (at the Zoo). Even if this Zookeeper’s favourite drink isn’t rum.

BUSY!

So Zoo 2 is live and we’re VERY busy at the moment – working on making things faster now….  

We are back!

As promised, back almost straight away…

Site back very soon!

We’re having a great response to the launch of Zoo 2, so we are having to move to a more powerful computer. It’ll only take a couple of minutes, and then everything will work faster!

Zoo 2 Launches!

The long wait is finally over, and anyone visiting the main Galaxy Zoo page will now see a brand new, sparkling Zoo 2 site. The results from the beta site are stunningly good, and we are sure that Zoo 2 will produce as much science – and as many surprises – as the original Galaxy Zoo did.

Of course, we need your help, so please do log on to the site and get classifying. Along the way you’ll notice a few new features; for starters, the images are hosted on our brand new server system at Amazon web services, so you should be able to classify as quickly as you like without us slowing you down. For the first time in Zoo 2, ‘My Galaxies’ works so you’ll be able to revisit galaxies gone by and find new information about old favourites. And speaking of favourites, you’ll be able to select and view your own favourite galaxies as you go.With that, I’ll say thank you to the team who put Zoo 2 together, particularly Arfon, and go and classify some galaxies.

P.S. The site’s up 90 minutes early, so if it’s before midnight GMT then don’t tell anyone ;-).

Twittering the day away

It’s hard to move, or at least to read a newspaper, in the UK without coming across a mention of Twitter, the microblogging service. If you’re interested, you can find our Twitter feed here – we seem to be counting down to something at the minute.

  • Auntie’s been on the phone : we’re at T minus 3 days and holding. 32 minutes ago
  • T minus 3 days… (this is getting exciting) 1 day ago
  • T minus 4 days… 2 days ago

follow me on Twitter