Dabr migration and Apache

After a recent purchase of VPS hosting from WebFusion, I wanted to move my high-traffic site dabr.co.uk onto it. With the site exceeding one million daily page views, I really had to be careful about how I’d move the site to it’s new home.

10%

Using the “Hostnames” report within Google Analytics, I was able to identify that 10% of users access the site via the m subdomain and 40% use www – the rest not using any subdomain at all. This was a real stroke of luck as it allowed me to test the new server with a controllable percentage of the total traffic, simply by updating the DNS records.

When given 10% load the VPS handled just fine, so after two days I moved on and tried it with 50%.

50%

A second report from Google Analytics also helped tremendously: hourly visits. With about 45% of hits coming from Indonesia (yes, really), I needed to know what time was the best window to try any drastic changes to the servers. The graph clearly shows that the longest quiet period I get is between 6pm and 9pm GMT – when most of Indonesia is asleep and web traffic drops in half:

On the first few occasions that I did try 50% of the load pointing at the new server – it buckled and almost completely fell over. A chat with Nick, a friend from Heavy Sumo, led me to explore the Apache configuration – and what a bizarre experience that was.

Here’s the settings that were used by default on the server, compared with the Apache defaults and the settings on the old server:

Setting WebFusion VPS Apache default Old server
StartServers 1 5 8
MinSpareServers 1 5 5
MaxSpareServers 5 10 20
MaxClients 10 256 256
MaxRequestsPerChild 0 1000 4000

Notice the difference?

I have no idea why WebFusion would have their defaults so low. Changing these to match the settings of the old server had a significant boost on performance.

I left 50% traffic on the new server for a couple of weeks, and tried a couple of times to add a bit more, but users quickly complained about the site slowing down.  The slow down matched up to the Indonesian peak time around 2pm GMT, so I rolled back the DNS to point to the old server again.

90%

This stage required a little further exploration of Apache performance tuning articles. One interesting fact I discovered was that Apache has a hard-coded MaxClients limit of 256, which you can only raise by recompiling Apache – so that wasn’t particularly an ideal option.

The Apache Performance Tuning article from devside.net was the answer to my problems. It suggested a number of different solutions, two of which were to adjust Keep Alive and Timeout settings.

Setting Webfusion VPS Devside
Timeout 300 45
MaxKeepAliveRequests 100 80
KeepAliveTimeout 15 2

These changes, combined with the forking settings above, have made the server run much more smoothly. In fact, it’s now taking 100% of the traffic and people are noticing an improvement in speed.

100%

With the new server running well for a month, I then asked the previous host to delete the old server. Hopefully the new server should last a while before I need to consider load balancing to a second server.

Posted in Uncategorized by David Carrington at March 17th, 2010.

Leave a Reply