farbot.com

It's taken a whole lot of trial and error and a little bit of internet research to finally identify what's wrong with my web server. I have had it stay offline quite a lot lately as I tried to diagnose the problem.

I like to tell people that I run my own server for three reasons. One, so I can keep copies of what I write elsewhere for myself for future reference, two, so I can run whatever software I want on it, and three, as a learning tool. These past few weeks have driven home the last function of the server several times over.

My last bout of tests (inspired by this Troubleshooting Memory Usage article) focused on memory utilization and sure enough it showed that the system was going off into la-la land when it started needing to use more and more swap space. A little while ago I upgraded the swap space on the machine, which removed the dire log messages indicating it was running out of memory, but that merely served to mask the problem. Once the system memory needs climb high enough to need a lot of swap space, performance grinds down to extremely low levels. It's not a malfunction, per se, it's just a system with higher requirements because of more software being run on it.

What was driving higher memory usage?
The nightly backup and verification process was causing available memory to drop to uncomfortably low levels. Once in that state, which was running for long periods of time as it worked on a server-wide backup, other activity would send the machine over the edge into an all-swap situation. At least that's my latest theory which recent changes seem to reinforce.

What fixed it?
I've done a few things along the long road of diagnosing the problem. Backups have been simplified. I've broken the backups down into much smaller and saner chunks. The vast majority of backed up data isn't changing -- the photo albums of years prior to this one are the prime consumers of space on the server -- so those can be backed up monthly, and I can kick off on-demand backups if I ever need to restructure them again. I did have to restructure the albums a bit to make it work, and write new shell scripts to simplify the backups. I learned a bit more about that along the way, so it's not so bad. I've also separated out the main system backup process from the album backup.

There are new scripts which now safely shut down the web server and database server entirely during the nightly backup process. The old nightly backup used to take hours on end, but with just the system and the current album year getting nightly differentials it took about 25 minutes in its latest run. This will of course expand, but it was fast enough most recently to fit entirely in a 30-minute SiteUptime.com check of the site. Not having the web server and database server running really cuts down memory usage and should make even the full monthly backups the incrementals are based on run safely.

What's left to do?
I need to port the shutdown script to the monthly backups and experiment with a possible way to redirect all sites to a "down for maintenance" page to avoid having the sites not respond at all.

How much memory does it have now?
The machine itself is sound, the drives aren't failing (knock on wood), and while the CPU fan might sound like it's seen better days, the core machine is a circa-2000 Pentium 4 running at 1.5 Ghz. Sounds like plenty of juice for a very low-hit web server, but memory is the other side of the coin. The old beast only has 384 Megs of RAM in it and while it is upgradeable it uses extremely rare RAMBUS memory that has always been heinously expensive. As a point of reference, a 1 GB stick of RAM for my current home desktop goes for $61. A 1 GB stick of RAM for that server goes for $400. Yeah, that's four hundred smackers.

I've also started getting more and more attention from search engine spiders as well as curious people clicking through my name link on Aeropause. The little spikes of activity from those few hits is enough to push the system into the red zone with so little memory when using a pretty hefty CMS engine like Drupal.

New hardware coming soon.
So it's time for me to buy a new core system with lots of RAM, a peppy CPU and a DVD-ROM reader would be nice, and then transplant the hard drives to it. I'll be trolling the deal boards for barebones systems, or hitting up friends for their older machines they want to upgrade.