Web performance, part 1

After experiencing some really heavy loads on the server last weekend (which caused some 503’s – sorry again) we pulled down some logs from the servers and applied some grep and sed magic on them. We were a little surprised by what we learned.

It turned out that over 20% of the requests to the servers were requests for a commit (Commits#show in Rails-speak). You guys seem to use Gitorious to read code! The thing is that all Git activity is quite expensive, so these 20-something percent requests accounted for 35% of the server’s rendering time.

The chance of a Git commit identified by a SHA changing over time is fairly low, which means it’s a very good candidate for caching. The only issue is that Gitorious displays quite a lot of personalized content, depending on whether you’re logged in or not: the top part of the page displays your username and unread message count, for instance. So if we naïvely cached it, the second user requesting a commit would receive the page rendered for the previous user viewing the same page.

So we extracted the diff and comment parts of the commit page into two separate parts, loaded using javascript when the page is loaded. This way, the full page is cheap to render, and the git data are loaded afterwards, in a payload that will be the same for every user.

Now all we had to do is to make sure these pages are rendered from cache. We’ve been running varnish in front of Apache on gitorious.org for quite a long time. Varnish uses some quite simple rules to determine whether contents can be cached or not. For our new pages this is what we needed to do to make Varnish serve them from the cache was:

• Set a Cache-Control header that tells Varnish for how long the content can be cached
• Make sure we’re not sending Set-Cookie headers from the Rails app, since Varnish by default will not cache data containing cookies

We just deployed this new code this afternoon, and have been watching the load on the server over the last few hours. Before we deployed it, the server was using all its Passenger processes to the full. After deploying the change, 2/3 of the processes have been idling, making Gitorious.org a whole lot snappier.

We’ll be putting more work into the performance of the web app over the next few weeks, so expect more posts like these in the feature. And as always, let us know if you’re having any issues, and we’ll help as good as we can!