- LittleBigPlanet Kratos / 007 Quantum of Solace Code Contest
- Is the GPhone a Trolltech Greenphone or Neo1973? [updated]
- EVE Cheaters Lead Gilded Lives
- Mobile: Opera Mini
- PSPWorld's Top 10 Shows Platform Weakness
- EA Lets You Try, Try Again Installing Spore
- AdhocParty Expands To Over 100 Titles
- Home Muzzled, GTA4 Still Great
- zOMG! It's Totally A Flash MMO!
- Save Room At The Table For Killzone 2
Farbot: Downtime
Wed, 2005-07-20 09:53 — PMunn
An ISP service interruption to my area throughout yesterday kept this site down. You can see on the front page that the server was brought back up yesterday evening in anticipation of a router reset, but that turned out not to be the problem.
As always, let me know if you see any site problems by using your email client or this handy form built into the site.
swap's not the whole story
The system became inaccessible this morning at the usual time, but I'm not seeing memory messages in the logs. Unfortunately I rebooted it without checking the screen.
I took a closer look at the logs, paying special attention to the apache logs since the downtime happens at a very similar time each day. I noticed that the mod_bw bandwidth throttler seems to causing massive amounts of thread deaths due to segfaults since I installed it back on 5/18.
I'm guessing the normal spidering activity from Google and others is what was causing it to die a terrible death.
I've disabled it, so we'll see how tomorrow goes.
swap increased
Last night the system fell down again. Given that all signs point to insufficient swap space for the system, I've increased it. Fortunately with Linux there are a couple of ways to do it.
One way is to use a swap partition, which is typically how new systems are set up. I was afraid I'd have to repartition and do this, but you don't have to do it this way.
The second way is using a swap file in an existing partition. As root, it's a very simple process, and I was done setting it up in about 60 seconds.
I used these instructions at RedHat's site. My system is now showing a gig more of swap than it used to have, so tune in tomorrow to find out if this fixes the problem.
crashed, again
We had a day or so of downtime. This was a combination of my ISP not being all that stable and my server itself falling down again. The crash looks almost identical to the one that happened in May according to the logs, so I'm going to need to dig deeper and see if I can read the logs or, more likely, find someone else who can make sense of them.
crashed
We had a couple of hours of downtime this morning, rougly from 5-7:30am. Something apparently caused the server to run out of memory and become unresponsive, effectively crashing.
I made a modification yesterday which I've rolled back in case it contributed to the memory problem.
cable modem switch
We were down for most of today on the standard ports while I got the cable modem changed out for a newer model that hopefully will run better.
The old model had the habit of rebooting itself on occasion, interrupting service and occasionally forcing my to cold-reboot the router.
At least that's my current theory. If this one has the same problem, I may need to have a technician come out and diagnose possible signal issues with whatever the modem is receiving through the coax wire in my house.
bit of downtime
There was a bit of downtime for the Gallery while I restored a month of images I'd accidentally erased. These backups are really coming in handy.
one day, not bad
We were down most of today, but nearly all site features are back online, near as I can tell. The proxy is down, but that's something I can fix tomorrow night.
good old encoding
We're back online with a much larger (and newer) hard drive. Welcome back.
It took me a little while to figure out that I had to recreate the phpbb database with SQL_ASCII encoding instead of UNICODE. It couldn't handle the special characters in some of the postings if I used the default, UNICODE database format.
disk full
Looks like my backup script malfunctioned and filled the disk up last night, severely hobbling the site.
I've cleaned up the immediate mess but will have to fix the backup script sometime tonight if I can.
Update: I fixed the backup script yesterday, and its first run very early this morning went well. Let's hear it for bash scripting.
connectivity
I've been suffering from odd connectivity issues that can either be my ISP or my router. The net result is that suddenly internet access fails, and I have to manually pull the power plug on both the cable modem and the router, then wait for first the one to come up before plugging in the other.
Sorry if you try to get here and the site's down. If you can, TXT me if you notice it's down and I'll check on it.
down for 3+ days
We were down for 3 days 8 hours because of a viral infection on the server, most likely brought in via scripting vulnerability in an earlier version of phpBB. Most of the server's functionality has been restored on a brand new install of Fedora Core 4 with Apache 2 and PHP 4.4.0.
Post new comment