Contrary to some rumors that may be out there I'm still alive, just popping my head up to see what's going on. As many know, ActiveRain had some, uh, pretty serious "technical issues" for a while which I've been helping to battle. Luckily things have stabilized quite a bit in the last week. I'll try to give you the post mortem on this one, warning tech talk ahead.
Traffic to ActiveRain's site has been steadily increasing month over month, I think we're averaging somewhere north of 80k visitors a day and 15M page views a month. Our servers were able to handle the human traffic fine, but then we started getting absolutely pounded by various bots and spiders trying to crawl the hundreds of thousands, if not millions, of pages on the site. For some reason they liked to all choose the exact same time to hit us too. Some of the RSS feed readers were particularly hideous, as unlike the major search engines that would throttle their requests and only try to say load a page a send, they would trying hitting every RSS feed on the site at the same time. This was causing traffic spikes that were large enough to start taking down our web servers periodically or at least make them so slow for periods of time they were nearly unusable.
About two months ago we ordered upgraded hardware doubling our number of web servers. Our managed hosting company built out this environment, but in the process upgraded almost all the software and operating system versions to the latest ones. As most computer users know, while upgrading software theoretically should help reduce problems it often has the opposite effect. The new environment went through our testing no problem, and seemed to function fine when we put it up. However, when we switched over, every couple hours our servers would just blow up (not litererally), almost at random. We spent many, many hours going line by line through log files, and doing everything we could to diagnose the issue. Not being able to find any reason for the random self destruction we began rolling software versions back methodically trying to find if one of those caused it.
None of the usual suspects seemed to help, finally we ended up rolling the version of linux we're running on back a minor version and the problem disappeared. Almost two weeks after the roll out of our new environent we had a fix. That was about a week ago, and things have greatly stabilized since then. We've got one more major hardware upgrade coming in the next day or two, as we're going to much more powerful database servers (4 core/4 GB of RAM to 16 core/32 GB ram) This should further speed up the site and give us a lot of buffer for future growth, and yes we are making sure they are all running the same software versions this time.
So that's what I've been up to and while all of my hair finally fell out :) No I'm not Jeff Turner's long lost twin...
;)
Links;)
Tags (Tag Cloud);)
Archives
Posted by 
I understand some of that where as I would not have a year ago. You have been missed but more importantly, we all appreciate you AR guys!