Today’s outage

Around 12:00 CET today, went down and was down for about 10 minutes. During this time, both the web site and git push/pull was unavailable. This is a short summary of what happened.

We have been having performance issues with our database over the last few months. The main bottleneck has been our events table, which keeps track of all kind of activity on The events table even contained one row for each commit ever pushed to, information that is already available through Git itself. A few weeks ago, we pushed a change to Gitorious which means we no longer create an entry in the database for each commit, only the push that occurs – and the push event keeps track of which commits were included. The commit data is then fetched from the Git log.

This morning we raised the amount of memory allocated to our MySQL server, specifically the innodb_buffer_pool_size setting, and restarted the database server. To be on the safe side of things we reloaded the Passenger instances, to keep them from hanging on to dead database connections. Initially, things seemed to be okay, until the web app went down about 15 minutes later. We tracked this down to about 100 Passenger instances waiting on the global queue, and found the MySQL server busy inserting data into the sessions table. By now the familiar sound of face-meets-palm was getting louder: we have a sessions table? Apparently, we did. Having one and not knowing it means it kept quite a lot of data too. So we switched to cookie based sessions and restarted the server, and things seem to be running smoothly now.

The reason why the database server stopped responding – blaming the huge amounts of data on ourselves – is that the file system where MySQL keeps its data went full while the server was coming back up after the restart. We’ll be adding more disk there now.

Sorry about the downtime, it happened because we’re addressing the performance on Changing things involve a risk of breaking them, which is what happened today.

One Comment

  1. Andreas
    Posted March 25, 2011 at 9:05 am | Permalink

    I assume you are already using innodb_file_per_table, seems to be the only way to ever shrink data size with innodb ^^

%d bloggers like this: